We have multiple clients and multiple UniVerse error logs. I wanted to create a dashboard similar to something like Sentry where we have all the errors in one central location. After thinking about it, I think I have a decent enough design to get started on working and it is half implemented now.
The first step is that I need agent scripts on each machine that will push the errlog file to an external server. I originally wanted this to be like ansible where there is no agent software. I wanted to write a script to ssh in, get the file and then end the connection. However this would involve juggling VPN connections and might be difficult. It would be easier to have things get pushed out rather than pulled in.
The second step once we have the errlog files on a server somewhere is to import them in.
The import script will pull the file to the server and then it process the file. The program will go line by line and hash each line. It checks to see if this line is already in the database. If it isn't in the database, parse the line into the specific fields and then update the database. I also save the raw line in the same table.
One optimization here would be to only send the lines that have changed. The agent script sends the entire errlog every few minutes and this is a bit wasteful.
The other solution would be to make the agent script smarter and have it remember the last line it sent and then only send each line when the line gets added to the file.
The central server could be a webserver that you can curl errors to. This would also be a good way to go about decreasing the amount of stuff that the server needs to do.
However the simplest form for now is to send the entire file and import it in.
The last part of this monitoring solution is to build a dashboard to view the errors in. I wrote it using express as the backend. It gets the data from the database and returns it. I then have a petite-vue frontend that will build the table.
I also added logic here that will refresh the data every few minutes and it will then change the colors of the newly added items so it is obvious when a new error arrives.
This should probably be a server side rendered page with some light interactivity but it feels weird to build out the table in ejs and then set up the json object in vue. This could also be a good chance to swap it out for something like htmx. I could also add in the express-vue package that seems to build out vue templates on the server.
As a proof of concept however I think I'm pretty happy with how it currently works. I can now quickly see all the errors in one page. The next step would be to add some visualizations because it would be cool to see errors broken up by type or broken up by client.