Creating a json object from one big php file or from alot of smaller php files?

phpn0w · July 7, 2019, 4:07pm

I get alot of data from different companies. I want to make a big JSON object which will contain all the data from all the companies togheter.

I am going to save the JSON object in a file instead of saving it in the DB.

My question is what would be best to do?

Lets say I get data from 50 different companies.

1. Have 1 single php-file for every company. open the json-file, insert the data and save it. Open next company-php-file, open the json-file and add this new data to it. Keep doing for 50 files.
2. Or create and insert all companies data in one single php-file. Hence only opening the file once per data update

JimL · July 7, 2019, 4:34pm

Are the company data the same, similar or different between the different companies? Also you can easily read the json file in one php file, then call 50 different php files to “do their thing”, and then write to the json file again to do both things you mention in op.

phpn0w · July 7, 2019, 5:08pm

I get the data in XML files and they dont look the same so I have to parser every company different but they have the exact same kind of data

benanamen · July 7, 2019, 5:13pm

With that many sources and data why would you not import it into a db? You can always output the data as JSON or whatever you want.

What happens when you have 100 sources? 500? It will easily become a maintenance nightmare.

phpn0w · July 7, 2019, 5:21pm

The reason to why I didnt want to store it in the DB is becasue I am so afraid of all the db connections and queries will make my website slow. I really want to make it as fast as possible. But when you put it like that I might go with a DB anyway

benanamen · July 7, 2019, 6:11pm

When you get to a place where you are able to slow down the database, come back and we can talk about Replication and Sharding.

phpn0w · July 7, 2019, 6:13pm

Okay thanks alot. But then I have pretty much the same question again. Should I parse every companies data and insert it into my db from a seperate php-file for each company or should it all be done from one php-file?

benanamen · July 8, 2019, 6:23am

There are several ways to go about it. You should look into ETL (Extract, Transform, Load). ETL is specifically for doing what you need to do. There are many software’s available both free and paid. The process can also be automated.

phpn0w · July 8, 2019, 11:04am

Would you recommend a software for doing that or something like PHP ETL github?