Bangle.js: Fetch activity data

joserebelo commented

2023-07-18 20:17:30 +00:00

Member

Implement activity fetching from the Bangle.js health app:

Add the rawIntensity column to the Bangle.js activity samples
Trigger activity fetching since the last fetch timestamp

TODO:

Set the device as busy correctly
Save last sync timestamp
Ensure the real-time samples do not over-count steps (floor the real-time samples down to the nearest 10 minutes?)
Testing / cross-check data

PR on BangleApps: https://github.com/espruino/BangleApps/pull/2889

Implement activity fetching from the Bangle.js health app: - Add the rawIntensity column to the Bangle.js activity samples - Trigger activity fetching since the last fetch timestamp TODO: - [x] Set the device as busy correctly - [x] Save last sync timestamp - [x] Ensure the real-time samples do not over-count steps (floor the real-time samples down to the nearest 10 minutes?) - [x] Testing / cross-check data PR on BangleApps: https://github.com/espruino/BangleApps/pull/2889

joserebelo added the

Bangle.js

label 2023-07-18 20:17:30 +00:00

joserebelo changed title from ~~Bangle.js: Fetch activity data~~ to WIP: Bangle.js: Fetch activity data

2023-07-18 20:18:46 +00:00

joserebelo commented

2023-07-18 20:26:50 +00:00

Author

Member

@gfwilliams I was playing around while trying to understand how the Bangle works, and actually missed d82ba7a04c, I looked around and could not find anything fetch-related..

There does not seem to be any code to handle it on BangleApps, but I can refactor this to work some other way.

@gfwilliams I was playing around while trying to understand how the Bangle works, and actually missed d82ba7a04c44050cc506f648e06569b6326351cb, I looked around and could not find anything fetch-related.. There does not seem to be any code to handle it on BangleApps, but I can refactor this to work some other way.

Ganblejs commented

2023-07-18 23:00:43 +00:00

Contributor

Could a situation come about where where we have a gap in data sent to gadgetbridge but the last fetch timestamp is newer then that gap so it doesn't get filled?

joserebelo commented

2023-07-18 23:26:27 +00:00

Author

Member

I guess that's possible. There's a button to set the last activity fetch timestamp in the debug activity, but we do need some logic to make this more resilient (eg. the bangle reporting how many items it sent, and if there's a mismatch, GB does not persist the new timestamp).

gfwilliams commented

2023-07-20 11:29:43 +00:00

Member

Thanks! Yes, I started this in Gadgetbridge and then didn't get around to the Bangle.js bit before my holiday.

For the timestamp, is it safer just to use what was done in d82ba7a04c - so looking at the last recorded activity item? So then even if a sync failed we don't risk getting duplicate items?

When connected normally the Bangle should send updates every 10 minutes so what we really care about is making sure that when we next reconnect we catch up with those.

In terms of doing long-term syncing (so catching up on data from before this PR gets merged), I'm not sure just having a single sync date var is good enough.

What I'm concerned about is that there will probably be some health records from when Gadgetbridge was connected (and the timestamp on those won't be exact as it's from when Gadgetbridge received the record), so when you do a sync right from the start of recorded time you'll end up with a whole bunch of duplicate records, which will then double the amount of recorded steps on all those days.

For a full sync, I think we want to iterate over all the health records Gadgetbridge has, and, when we see any records that are > ~15 minutes apart we ask Bangle.js to search and fill in the time period inbetween?

Thanks! Yes, I started this in Gadgetbridge and then didn't get around to the Bangle.js bit before my holiday. For the timestamp, is it safer just to use what was done in https://codeberg.org/Freeyourgadget/Gadgetbridge/commit/d82ba7a04c44050cc506f648e06569b6326351cb - so looking at the last recorded activity item? So then even if a sync failed we don't risk getting duplicate items? When connected normally the Bangle should send updates every 10 minutes so what we really care about is making sure that when we next reconnect we catch up with those. In terms of doing long-term syncing (so catching up on data from before this PR gets merged), I'm not sure just having a single sync date var is good enough. What I'm concerned about is that there will probably be some health records from when Gadgetbridge was connected (and the timestamp on those won't be exact as it's from when Gadgetbridge received the record), so when you do a sync right from the start of recorded time you'll end up with a whole bunch of duplicate records, which will then double the amount of recorded steps on all those days. For a full sync, I think we want to iterate over all the health records Gadgetbridge has, and, when we see any records that are > ~15 minutes apart we ask Bangle.js to search and fill in the time period inbetween?

joserebelo commented

2023-07-22 09:35:51 +00:00

Author

Member

@gfwilliams just a couple of statements/questions to ensure my assumptions are correct:

The bangle currently sends an activity/steps sample every 10 minutes
The samples are persisted to the Bangle health database at those 10 minutes
The timestamp in Gadgetbridge might be off by a bit, since we're using the current time from the phone. Since we sync time with the watch, it should not be off by a lot.
If we query the bangle health database in-between, what do we get?
1. A partial sample?
2. No sample yet persisted?
What happens when the user triggers a heart rate measurement in an interval < 10 minutes? Does only the latest sample get persisted? Do we send it immediately to the phone?

For a full sync, I think we want to iterate over all the health records Gadgetbridge has, and, when we see any records that are > ~15 minutes apart we ask Bangle.js to search and fill in the time period inbetween?

I was thinking it might be easier / more robust to request the full database at first sync (last sync timestamp in Gadgetbridge == 0), then reconcile the entire database. It should be a one-time thing, but it's not yet clear if it would take too long for users that have had the bangle for a while. This would allow us to sync samples older than the earliest sample we get on Gadgetbridge too.

In terms of doing long-term syncing (so catching up on data from before this PR gets merged), I'm not sure just having a single sync date var is good enough.

I am not sure I get this. Are you referring to the case where in the long-term we continue sending the real-time samples, and we still get gaps in the data?

@gfwilliams just a couple of statements/questions to ensure my assumptions are correct: 1. The bangle currently sends an activity/steps sample every 10 minutes 2. The samples are persisted to the Bangle health database at those 10 minutes 3. The timestamp in Gadgetbridge might be off by a bit, since we're using the current time from the phone. Since we sync time with the watch, it should not be off by a lot. 3. If we query the bangle health database in-between, what do we get? 1. A partial sample? 2. No sample yet persisted? 4. What happens when the user triggers a heart rate measurement in an interval < 10 minutes? Does only the latest sample get persisted? Do we send it immediately to the phone? > For a full sync, I think we want to iterate over all the health records Gadgetbridge has, and, when we see any records that are > ~15 minutes apart we ask Bangle.js to search and fill in the time period inbetween? I was thinking it might be easier / more robust to request the full database at first sync (last sync timestamp in Gadgetbridge == 0), then reconcile the entire database. It should be a one-time thing, but it's not yet clear if it would take too long for users that have had the bangle for a while. This would allow us to sync samples older than the earliest sample we get on Gadgetbridge too. > In terms of doing long-term syncing (so catching up on data from before this PR gets merged), I'm not sure just having a single sync date var is good enough. I am not sure I get this. Are you referring to the case where in the long-term we continue sending the real-time samples, and we still get gaps in the data?

gfwilliams commented

2023-07-24 09:31:05 +00:00

Member

The bangle currently sends an activity/steps sample every 10 minutes

Yes.

The samples are persisted to the Bangle health database at those 10 minutes

Yes.

The timestamp in Gadgetbridge might be off by a bit, since we're using the current time from the phone. Since we sync time with the watch, it should not be off by a lot.

Exactly - I haven't checked, but I'd say a few seconds - definitely less than a minute!

If we query the bangle health database in-between, what do we get? A partial sample?

The file in storage is the last full sample (so the previous 10 minute window). You can use Bangle.getHealth... to get the current one but we don't want to do that normally as it won't contain complete data.

What happens when the user triggers a heart rate measurement in an interval < 10 minutes?

In 'realtime' mode (https://codeberg.org/Freeyourgadget/Gadgetbridge/src/branch/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/service/devices/banglejs/BangleJSDeviceSupport.java#L1290-L1292) you get much more often data reported to you, and the Bangle's 'health' library isn't involved and doesn't save that more detailed info on the Bangle either.

As you note in the code comment, it'll cause step over-counting so I guess we need to keep track of realtime steps and then subtract them from the 'health' event we get. Maybe Bangle.js needs to add rt:true to realtime data to make it easier (maybe it should even be a different event name?).

request the full database at first sync

Assuming someone's had their Bangle for a year (many had it since the KickStarter so getting on for 2 years now): 6/hr * 24 * 365 * 20 bytes = 1mb transfer

The actual transfer speed depends a bit on the device, but we're looking at probably over a minute of tranfers? It's quite a lot to block the device for (especially if there is no indication shown) and I don't think is something we'd want to do automatically on first connect as it could provide quite a bad experience.

long-term syncing

I just meant someone might have run Gadgetbridge for a while previously so they already have at least some data in it, but then they upgrade and get this code and they sync. In that case we'll be sending a lot of data that has already been gathered.

Plus if someone deleted their Gadgetbridge health data somehow (is it even possible) if the sync date hadn't been reset they would have no way of re-syncing to get the data back.

Anyway, it's not a big deal - I just felt like having Gadgetbridge being able to figure out where it had gaps in the data would be neat, but I guess it has issues - if the user routinely turns their Bangle off then there might be many gaps that Gadgetbridge has to request to be filled in.

We just have to watch out for de-duplicating data in Gadgetbridge - not accepting any duplicate data that arrives within 1 minute of each 10 minute chunk (unless it's realtime).

> The bangle currently sends an activity/steps sample every 10 minutes Yes. > The samples are persisted to the Bangle health database at those 10 minutes Yes. > The timestamp in Gadgetbridge might be off by a bit, since we're using the current time from the phone. Since we sync time with the watch, it should not be off by a lot. Exactly - I haven't checked, but I'd say a few seconds - definitely less than a minute! > If we query the bangle health database in-between, what do we get? A partial sample? The file in storage is the last full sample (so the previous 10 minute window). You can use `Bangle.getHealth...` to get the current one but we don't want to do that normally as it won't contain complete data. > What happens when the user triggers a heart rate measurement in an interval < 10 minutes? In 'realtime' mode (https://codeberg.org/Freeyourgadget/Gadgetbridge/src/branch/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/service/devices/banglejs/BangleJSDeviceSupport.java#L1290-L1292) you get much more often data reported to you, and the Bangle's 'health' library isn't involved and doesn't save that more detailed info on the Bangle either. As you note in the code comment, it'll cause step over-counting so I guess we need to keep track of realtime steps and then subtract them from the 'health' event we get. Maybe Bangle.js needs to add `rt:true` to realtime data to make it easier (maybe it should even be a different event name?). > request the full database at first sync Assuming someone's had their Bangle for a year (many had it since the KickStarter so getting on for 2 years now): 6/hr * 24 * 365 * 20 bytes = 1mb transfer The actual transfer speed depends a bit on the device, but we're looking at probably over a minute of tranfers? It's quite a lot to block the device for (especially if there is no indication shown) and I don't think is something we'd want to do automatically on first connect as it could provide quite a bad experience. > long-term syncing I just meant someone might have run Gadgetbridge for a while previously so they already have at least some data in it, but then they upgrade and get this code and they sync. In that case we'll be sending a lot of data that has already been gathered. Plus if someone deleted their Gadgetbridge health data somehow (is it even possible) if the sync date hadn't been reset they would have no way of re-syncing to get the data back. Anyway, it's not a big deal - I just felt like having Gadgetbridge being able to figure out where it had gaps in the data would be neat, but I guess it has issues - if the user routinely turns their Bangle off then there might be many gaps that Gadgetbridge has to request to be filled in. We just have to watch out for de-duplicating data in Gadgetbridge - not accepting any duplicate data that arrives within 1 minute of each 10 minute chunk (unless it's realtime).

joserebelo commented

2023-07-24 09:57:10 +00:00

Author

Member

Exactly - I haven't checked, but I'd say a few seconds
...
The file in storage is the last full sample (so the previous 10 minute window).

Okay, cool - this makes it easier.

so I guess we need to keep track of realtime steps and then subtract them from the 'health' event we get.

Yup... but this is starting to get more complex than I would have otherwise thought and like :)

I don't think that in Huami devices we persist the real-time steps, we just broadcast them for the real-time steps activity data. Should we stop persisting them? Then we will just have to worry about the gaps in the data going forward (we still need to take them into account for the first sync).

The actual transfer speed depends a bit on the device, but we're looking at probably over a minute of tranfers? It's quite a lot to block the device for (especially if there is no indication shown) and I don't think is something we'd want to do automatically on first connect as it could provide quite a bad experience.

This is a good point. This does not happen on first connect, activity data syncing is a separate operation, and there is an indication in Gadgetbridge, but not on the device.

Plus if someone deleted their Gadgetbridge health data somehow (is it even possible) if the sync date hadn't been reset they would have no way of re-syncing to get the data back.

There is currently a button in the debug activity to set the activity fetch timestamp, so it's possible to go back and re-sync.

In the worst cases, I'm expecting there to be a lot of gaps on the data, requesting these one by one and ensuring that everything is consistent at the end will be tricky, I need to think about this a bit more.

Follow-up questions:

(it's not clear to me from the code): does the bangle send these 10-minute updates when the values are zero?
If we enable real-time data, does the 10-minute sample that gets sent later exclude these previously sent steps?

> Exactly - I haven't checked, but I'd say a few seconds > ... > The file in storage is the last full sample (so the previous 10 minute window). Okay, cool - this makes it easier. > so I guess we need to keep track of realtime steps and then subtract them from the 'health' event we get. Yup... but this is starting to get more complex than I would have otherwise thought and like :) I don't think that in Huami devices we persist the real-time steps, we just broadcast them for the real-time steps activity data. Should we stop persisting them? Then we will just have to worry about the gaps in the data going forward (we still need to take them into account for the first sync). > The actual transfer speed depends a bit on the device, but we're looking at probably over a minute of tranfers? It's quite a lot to block the device for (especially if there is no indication shown) and I don't think is something we'd want to do automatically on first connect as it could provide quite a bad experience. This is a good point. This does not happen on first connect, activity data syncing is a separate operation, and there is an indication in Gadgetbridge, but not on the device. > Plus if someone deleted their Gadgetbridge health data somehow (is it even possible) if the sync date hadn't been reset they would have no way of re-syncing to get the data back. There is currently a button in the debug activity to set the activity fetch timestamp, so it's possible to go back and re-sync. In the worst cases, I'm expecting there to be a lot of gaps on the data, requesting these one by one and ensuring that everything is consistent at the end will be tricky, I need to think about this a bit more. Follow-up questions: 1. (it's not clear to me from the code): does the bangle send these 10-minute updates when the values are zero? 2. If we enable real-time data, does the 10-minute sample that gets sent later exclude these previously sent steps?

gfwilliams commented

2023-07-24 10:15:46 +00:00

Member

Ok, sounds good having the last sync date then...

we just broadcast them for the real-time steps activity data. Should we stop persisting them?

Yes, that sounds like a great idea! So we just need an rt:true in https://github.com/espruino/BangleApps/blob/master/apps/android/boot.js#L196

does the bangle send these 10-minute updates when the values are zero?

Yes

If we enable real-time data, does the 10-minute sample that gets sent later exclude these previously sent steps?

No - so best to just mark the realtime data as realtime and ignore it on Gadgetbridge - should be backwards compatible too

Ok, sounds good having the last sync date then... > we just broadcast them for the real-time steps activity data. Should we stop persisting them? Yes, that sounds like a great idea! So we just need an `rt:true` in https://github.com/espruino/BangleApps/blob/master/apps/android/boot.js#L196 > does the bangle send these 10-minute updates when the values are zero? Yes > If we enable real-time data, does the 10-minute sample that gets sent later exclude these previously sent steps? No - so best to just mark the realtime data as realtime and ignore it on Gadgetbridge - should be backwards compatible too

joserebelo commented

2023-07-24 20:41:00 +00:00

Author

Member

@gfwilliams what if we start syncing from the current timestamp once this is released in Gadgetbridge, but allow the user to trigger a full sync from the device preferences? Clicking it would warn the user that it might take a while, as well as display the progress. GB would fetch the full data from the Bangle, and persist in the Gadgetbridge database as needed, taking into account any gaps or already existing data.

gfwilliams commented

2023-07-25 10:36:06 +00:00

Member

what if we start syncing from the current timestamp once this is released in Gadgetbridge, but allow the user to trigger a full sync from the device preferences?

That sounds perfect to me, thanks!

> what if we start syncing from the current timestamp once this is released in Gadgetbridge, but allow the user to trigger a full sync from the device preferences? That sounds perfect to me, thanks!

joserebelo force-pushed jr-bangle-activity from 7219dd17af to 795a44c748

2023-07-27 19:47:38 +00:00

Compare

Ganblejs commented

2023-07-31 20:38:47 +00:00

Contributor

Closes #3133 I believe

👍 1

gfwilliams commented

2023-08-03 09:20:19 +00:00

Member

Looks good to me! Is sampleBuffer needed? It seems to be defined in the last commit but I don't see it used.

Also, is onFetchRecordedData called automatically when the Bangle is connected? It would be really good to be able to ensure that without having to click 'fetch', if there were activity samples while the Bangle was disconnected, they are automatically synced (at least if it's just a day's worth or so that'll be caught up in a fraction of a second)

Looks good to me! Is `sampleBuffer` needed? It seems to be defined in the last commit but I don't see it used. Also, is `onFetchRecordedData` called automatically when the Bangle is connected? It would be really good to be able to ensure that without having to click 'fetch', if there were activity samples while the Bangle was disconnected, they are automatically synced (at least if it's just a day's worth or so that'll be caught up in a fraction of a second)

joserebelo commented

2023-08-03 09:22:01 +00:00

Author

Member

@gfwilliams this is still incomplete, i still need to work on the de duplication. I'll try and finish in a couple of days once I'm back.

I pushed some work-in-progress code, the sample buffer will eventually be needed, but not yet being used.

@gfwilliams this is still incomplete, i still need to work on the de duplication. I'll try and finish in a couple of days once I'm back. I pushed some work-in-progress code, the sample buffer will eventually be needed, but not yet being used.

joserebelo force-pushed jr-bangle-activity from 38d8b8b271 to 74e5f8d369

2023-08-05 00:01:22 +00:00

Compare

joserebelo commented

2023-08-05 00:05:46 +00:00

Author

Member

Updated PR to match the latest changes in BangleApps.

I have not yet tested this extensively, but should be working and at least sync new data.

It should now only be missing deduplication of samples and testing.

Is sampleBuffer needed? It seems to be defined in the last commit but I don't see it used.

I was thinking of buffering the data, cross-checking the count of samples at the end, and not persist the timestamp if there is a mismatch (eg. some packet failed to be parsed or lost). I will be removing this for now.

Also, is onFetchRecordedData called automatically when the Bangle is connected?

Not right now, it is either called:

Manually, when clicking the sync button on the device card or Gadgetbridge notification
On phone unlock, by enabling "Auto fetch" in the settings
Through intents, using the intent API

However, I don't see why we could not call this on connection, so I will add that.

Updated PR to match the latest changes in BangleApps. I have not yet tested this extensively, but should be working and at least sync new data. It should now only be missing deduplication of samples and testing. > Is sampleBuffer needed? It seems to be defined in the last commit but I don't see it used. I was thinking of buffering the data, cross-checking the count of samples at the end, and not persist the timestamp if there is a mismatch (eg. some packet failed to be parsed or lost). I will be removing this for now. > Also, is onFetchRecordedData called automatically when the Bangle is connected? Not right now, it is either called: - Manually, when clicking the sync button on the device card or Gadgetbridge notification - On phone unlock, by enabling "Auto fetch" in the settings - Through intents, using the intent API However, I don't see why we could not call this on connection, so I will add that.

joserebelo force-pushed jr-bangle-activity from 74e5f8d369 to 8c9c656607

2023-08-05 14:58:51 +00:00

Compare

joserebelo changed title from ~~WIP: Bangle.js: Fetch activity data~~ to Bangle.js: Fetch activity data

2023-08-05 15:00:21 +00:00

joserebelo changed title from ~~Bangle.js: Fetch activity data~~ to WIP: Bangle.js: Fetch activity data

2023-08-05 15:06:39 +00:00

joserebelo commented

2023-08-05 15:07:30 +00:00

Author

Member

Added de-duplication of samples, this PR should be mostly ready. However, the realtime samples are offset by 10 minutes when compared with the duplicates. Discussion started in the BangleApps PR.

gfwilliams commented

2023-08-07 09:05:12 +00:00

Member

Great! Thanks for all your work on this.

I guess this is good to go now - the 10 min offset is something that we should definitely handle on the bangle.js side

Great! Thanks for all your work on this. I guess this is good to go now - the 10 min offset is something that we should definitely handle on the bangle.js side

joserebelo force-pushed jr-bangle-activity from 8c9c656607 to d4b4cb6a07

2023-08-08 21:01:23 +00:00

Compare

joserebelo changed title from ~~WIP: Bangle.js: Fetch activity data~~ to Bangle.js: Fetch activity data

2023-08-08 21:08:41 +00:00

joserebelo force-pushed jr-bangle-activity from d4b4cb6a07 to a95820d09e

2023-08-08 21:11:17 +00:00

Compare

joserebelo merged commit a95820d09e into master

2023-08-09 08:08:45 +00:00

joserebelo deleted branch jr-bangle-activity

2023-08-09 08:08:46 +00:00