Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split apps.json #1020

Closed
qucchia opened this issue Dec 7, 2021 · 21 comments
Closed

Split apps.json #1020

qucchia opened this issue Dec 7, 2021 · 21 comments

Comments

@qucchia
Copy link
Contributor

qucchia commented Dec 7, 2021

It seems a little counter intuitive to me that every app has its own folder for code, image, README, and yet their actual data is outside in a file called apps.json. This file is already quite large, and will probably not scale well.

My suggestion is to split this file into many files called app.json (or another sensible name) inside each of the apps' folders. We'd have to automate this process as there are a lot of apps, but it wouldn't be too difficult. Then at runtime these can all be put together into one file automatically.

@elrod16
Copy link
Contributor

elrod16 commented Dec 7, 2021

I would second this because I've already ran into trouble with trying to edit apps.json and have had it exhaust my browser's RAM.

@maybeanerd
Copy link
Contributor

maybeanerd commented Dec 7, 2021

In theory, we should not need the apps.json at all.
The app loader could look at all the folders it has, scrape them for their app.json, and give IDs based on the path (which afaik is forced to be the same as the ID currently anyways)

edit: just now realized this could be implied by your suggestion, but I read it as "keep apps.json as a list of IDs". If so, ignore this message and just count this as a +1

@elrod16
Copy link
Contributor

elrod16 commented Dec 7, 2021

The only issue I could see would be metadata like app category and which bangle it is for.

@jeroenpeters1986
Copy link
Contributor

I like this idea too, and started to think about it, but at the moment I wouldn't know how to loop over the subdirectories with native JS. Currently in core/js/index.js, apps.json is explicitly given as a file to check and is handled with XMLHttpRequest etc.

@maybeanerd
Copy link
Contributor

The only issue I could see would be metadata like app category and which bangle it is for.

That information should be included in their respective app.jsons, right?

@elrod16
Copy link
Contributor

elrod16 commented Dec 7, 2021

I'm referring to the proposal of eliminating any app.json

@maybeanerd
Copy link
Contributor

I'm referring to the proposal of eliminating any app.json

That's not what I meant to propose, sorry for the confusion 😅

@gfwilliams
Copy link
Member

This has actually been asked several times.

ran into trouble with trying to edit apps.json and have had it exhaust my browser's RAM.

It's a 160k file, so not huge at all - so I so wonder if your browser needs looking at :)

The app loader could look at all the folders it has, scrape them for their app.json

Have you considered how slow that might be, not to mention the load on the server? Someone goes to the app loader on a mobile network and then it has to do over 300 HTTP requests before it can even display the names of every app?

Also it's not possible to do a directory search when hosted on GitHub pages, so you'd have to have some specific code to use the GitHub API. If you're on a webserver where directory listing is disabled (like many) running the app loader would not be an option at all.

There's a bit of detail on it in this issue: #46

So really I think there are two solutions here:

  • Inconvenience literally every user with huge load times because you want to save yourself a bit of trouble when adding new apps.
  • We have an offline build process where GitHub pages-hosted sites are super slow, but the banglejs.com/apps one isn't because it creates its own big apps.json offline. I've already had a lot of push back about stuff like that - eg for reviews

@jeroenpeters1986
Copy link
Contributor

The only thing I can think of, but what isn't great, is that you have an app.json, manifest or whatever document per app, and on-commit (or on the server CI/CD wise) it will make the apps.json and commits it into the repo.
Is that what you mean with your last point too @gfwilliams ?

Personally, I don't think this would be ideal for all of the users, as you introduce a new process at a very crucial point, which can break and might not run on everyones computer or environment. In your docs where you explain to fork this repo and just WebIDE edit the apps, this would already create an impediment I think.

Also, I wouldn't know why one would eliminate all the apps.json, bcause that is where the meta information is. If that really created RAM problems in the computer/browser I assume it would be a user error somewhere in the editing. Which is also covered in the docs (make sure to have all the brackets)

@gfwilliams
Copy link
Member

Is that what you mean with your last point too @gfwilliams ?

Yes and no... I think if there's an auto-generated file (like apps.json would be) I don't think it should really be in the repo, as it means pretty much every commit is going to be followed by a machine-generated one. If we did this I think I'd have a script that you could run to create apps.json if you're hosting yourself and you have Node.js, but otherwise it'd just have to work by trying to load the 300 files from within the browser.

@elrod16
Copy link
Contributor

elrod16 commented Dec 8, 2021

It's a 160k file, so not huge at all - so I so wonder if your browser needs looking at :)

While the file itself is small, the github tab with the editor open is a huge resource hog, especially if you're traveling light (tablet/netbook).
While using an external editor is possible it can make a mess out of commits.

@jeroenpeters1986
Copy link
Contributor

jeroenpeters1986 commented Dec 8, 2021

Yes and no... I think if there's an auto-generated file (like apps.json would be) I don't think it should really be in the repo, as it means pretty much every commit is going to be followed by a machine-generated one. If we did this I think I'd have a script that you could run to create apps.json

Then I understand not splitting it at the moment, because you probably need all kinds of node/npm stuff (or just a simple python script) to run it, and this would require a local development setup.

While the file itself is small, the github tab with the editor open is a huge resource hog, especially if you're traveling light (tablet/netbook).

I don't think the development process should be altered so heavily for solely this reason

@elrod16
Copy link
Contributor

elrod16 commented Dec 8, 2021

I don't think the development process should be altered so heavily for solely this reason

True and I didn't propose removing it, but I think the resource issue needs to be kept on the mental back burner because as already mentioned above, that file is only going to get bigger.

Not to mention that with the intended goal of this project being an open and inclusive platform with a low entry threshold; it is important to remember not everyone has access to awesome computers. Even more so in the face of the chip shortages.

@hughbarney
Copy link
Contributor

we definitely dont want to do anything that will make a developers github hosted App Loader any slower to update than it already is. I'm probably the odd one out here in that I do my development on a repo fetched and merged from the master on a linux environment on my Chromebook. Only when I think my App is ready do I push it to my github repo to test the App loader and apps.json setup. But once I have done the push it can take 5-10 minutes before everything is properly reflected on the App Loader page. I would not want to have to setup additional tools locally like NodeJs etc. BUT with more novice users coming along who dont understand merges in GIT then having an app-specific app.json file in each directory is probably less likley to end up with merge issues.

@gfwilliams
Copy link
Member

Merge issues are a pain - I wonder if there's some kind of Git setting that can ensure it uses a more sensible (JSON-aware) way of merging the file than line-by-line so it 'just works'.

@SaraVieira
Copy link

I was working on editing an app and ran into slowness caused by VSCode trying to run prettier on that file

My idea would be to use the type of divide the apps that current exist and have a index.js on that folder that imports and exports each of the file of apps, something like so:

apps/
├─ Index.js
├─ clocks.json
├─ widgets.json
├─ ....

There would still be some more requests but it would be doable for the user in my opinion

Another option is to host it somewhere else like vercel or netlify that are also always free for open source and can run a build script

@rigrig
Copy link
Contributor

rigrig commented Dec 17, 2021

Do people actually want to run their own full app store, or only test/load their new app?
Maybe the store could try to load apps.json, and if that fails ask for a appid to load?
That way users can use GitHub pages to test their app, and the official store can use apps.json to list all apps.

@gfwilliams
Copy link
Member

@SaraVieira not sure I understand? You mean the index.js would reference each of the sub-json files?

I'd be happy splitting by widget/clock/app/etc - it seems a good middle-ground and helps to keep the overall file size down. However I don't think it'd appease those that want each app to have its own JSON file.

Do people actually want to run their own full app store, or only test/load their new app?

Some do, yes... There are some companies, but also even some using non-Bangle.js watches that want to ensure every app works with it.

adamschmalhofer added a commit to adamschmalhofer/BangleApps that referenced this issue Jan 4, 2022
Reading

espruino#1020
espruino#46

My understanding is that the requirement for a splitt apps.json solution
is:

  - work seemlessly with github pages (no manually calling a script)

  - single (or at most a few) apps.json for the browser to load for the
    store so the store page is fast

As the github pages already use jekyll to build the store page this can
be solved by a minimal jekyll plugin. This is a proof of concept of
that. If the concept is accepted in a next step, I will create the
splitt files for mmore that just the two here.
adamschmalhofer added a commit to adamschmalhofer/BangleApps that referenced this issue Jan 5, 2022
Reading

espruino#1020
espruino#46

My understanding is that the requirement for a splitt apps.json solution
is:

  - work seemlessly with github pages (no manually calling a script)

  - single (or at most a few) apps.json for the browser to load for the
    store so the store page is fast

As the github pages already use jekyll to build the store page this can
be solved by a minimal jekyll plugin. This is a proof of concept of
that. If the concept is accepted in a next step, I will create the
splitt files for mmore that just the two here.
@gfwilliams
Copy link
Member

Forum post to ask for input on this: http://forum.espruino.com/conversations/372018/

@Ronin0000
Copy link
Contributor

Should this be closed?

@gfwilliams
Copy link
Member

Thanks! Yes, definitely now each app has metadata.json :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants