Analysing Log Data Efficiently

A little bit of background: I have built a microcontroller-based system to run my heating and hot water system (solar tubes, underfloor heating etc) which works well. This new system replaces an old system written in VB6 which was very reliable, but very untidy, hardware-wise - cables and modules all over the place! The new system is very compact.

The one thing that has been missing is the ability to see what has been happening with all the temperature sensors, pumps, valves etc. The old system used a very clever program written by a guy called Nick Hubbard. This program would take a simple text file, in which each line comprised a time of day, sensor number and temperature, then display these on a chart on the PC. It was incredibly fast and very flexible, allowing zooming and formatting.

This was also useful because I could access it very easily from anywhere in the world simply by using a VNC program. The new system, however, doesn’t use PCs at all. The controller part is linked to a NodeMCU ESP8266 which allows access to the system via the web. However, with very limited storage (a typical day’s data file is around 900K in size) I have decided to write logging data (sensor temperatures and system activities) to a MySQL database on my ISP’s server. This is working very well.

I did think about trying to recreate the Hubbard display program in PHP, but that’s a bit beyond my present capabilities! As an alternative I have written a PHP routine which reads the MySQL database and creates a text file in the identical format to the old system. I can then download this and open it with the program and bingo! I have my historical display of data.

The purpose of this question (you knew I would get there eventually!) is to ask if there is any simple way to do the last part of this procedure. At the moment, I run the PHP program by calling it up in my browser. I then have to use an FTP program to download the text file to my PC and then open the text file with the Display program. There has to be a better way.

I have avoided technical details at this stage as I didn’t want to write even more than I have. However, I am very happy to provide any details to anyone that thinks they might have a good idea as to how to make this whole process more efficient.

Thanks in advance!

PM me if needed.

Rather than using ftp, would just going to the site and clicking the file be acceptable? Or are you really looking to automate it where you just open the program?

Thanks for coming back to me. As you say, I can get the file into my browser, but then I still need to copy and paste it into a file to be opened by the Display program. What I would REALLY like to do is emulate the program in PHP so that I simply visit the web site and have the latest info displayed on-screen. Even if I am willing to do away with such features as zooming in, I’d still need a means of ‘building’ an image pixel by pixel, using the sensor data in the file, and I’m really not sure how one would go about that!

This is the kind of thing I am after. It may look a complete mess, but it shows me exactly what I need to know!

Then I would look into the d3.js library, https://observablehq.com/@d3/gallery

How about installing a localhost xAMP (Apache, Mysql, Php) system on your PC and just do everything there, using scheduled tasks/cron jobs to automate the gathering of data and generation of the files that the old graphic software uses?

1 Like

Thank you - that looks VERY comprehensive and impressive! It’ll take a while to get to grips with, though!

The problem is that I want to be able to access this from the UK - the system is in Spain! But I think I have found a solution which, though not perfect, is pretty good - see below.

Thanks to those who have suggested solutions. Much to my surprise (should I be surprised?) I have found that if I enter the URL of the file created on the server into the Display program, after a very brief pause it opens and shows the data perfectly! I had never realised that I could do that, and it solves most of my problems.

1 Like
Sponsor our Newsletter | Privacy Policy | Terms of Service