9af3115cab
fix: no listens are being crawled ( #306 )
...
The supervisor job handler was accidentally disabled in a previous
commit.
2023-10-08 15:56:09 +02:00
7140cb0679
feat: import listens from spotify extended streaming history ( #305 )
2023-10-01 03:35:02 +02:00
625db7dbe7
fix: slow query taking up 66% of db time ( #298 )
...
Query was scanning the full index on listen table,
when we really only need data from the last hour
2023-09-18 21:15:37 +02:00
9ba47a560c
fix: invalid database migration
2023-09-17 00:43:41 +02:00
dd57a52ab6
feat: optimize db queries ( #297 )
2023-09-17 00:31:39 +02:00
89440daf7b
feat: improve listens report response time
...
Reduce response time for larger intervals (e.g. all time) from 9 seconds
to 150ms in my tests.
2023-09-16 23:20:15 +02:00
38cf2ff549
chore(deps): bump all ( #294 )
2023-09-16 13:02:19 +02:00
renovate[bot]
45f8d762c2
chore(deps): update opentelemetry-js monorepo ( #252 )
...
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Julian Tölle <julian.toelle97@gmail.com>
2023-09-16 12:14:45 +02:00
689868798d
fix: failing healthcheck for spotify api
...
API started returning 403 on all requests
2023-09-10 20:59:58 +02:00
bbc7ec0817
fix: missing import for @hapi/joi ( #274 )
...
Replaced the module in the dependencies but not in actual usage
2023-05-07 13:36:55 +02:00
8721fd101d
feat: use bulk Spotify API calls & inserts to make spotify crawling more efficient ( #271 )
...
* feat(spotify-api): bulk endpoints
* feat(music-library): allow bulk operations
* feat(spotify): bulk track+album+artist+genre import
* feat(spotify): use bulk import api for user crawl
* feat(spotify): bulk listen insert
For the benchmark case of a new user where Listory imports 50 new listens along with all now tracks, artists, albums & genres we significantly reduced the number of things happening:
Spotify API Requests: 208 => 8
DB Insert: 96 => 8
Tracing Spans: 1953 => 66
2023-05-07 02:20:43 +02:00
107a81412f
refactor: remove unused logging setup
...
Pino is actually set up in src/logger/logger.ts and the variable
created here is never used.
2023-04-02 13:16:11 +02:00
Marc Mogdanz
5ce51c389c
fix(api): use PORT var ( #262 )
2023-03-19 16:02:49 +01:00
0e2c56bf5f
fix(api): properly handle errors in crawler loop
2023-03-17 17:37:14 +01:00
471f46eb8d
fix(api): album&track reports only contain one artist
...
Albums & Tracks can have multiple artists associated with them, but
the previous SQL queries would only return a randomly-chosen one of them.
2023-03-17 17:37:14 +01:00
821b91defe
fix(api): importer job was always importing first user
...
We did read the actual user id from the job data, which caused us
to always import the first user returned by the database.
2023-03-12 21:33:13 +01:00
46b1650066
feat(api): import listens after first sign in
...
Previously it took a minute until the import was started within the normal
loop. This way, we create an import job right after the user logs
in for the first time.
2023-03-12 02:20:39 +01:00
14478a5418
feat(api): poll listens less often if user is inactive
...
To save on Spotify API requests we have two different classes of
polling intervals:
- all users are polled at least every 10 minutes, this is a safe interval
and no listens will be ever missed
- if a user listened to a song within the last 60 minutes, we poll every
minute to ensure that the UI shows new listens immediately
2023-03-12 02:20:39 +01:00
b9f92bbdfa
feat(api): use nest-pg-boss for spotify interactions
2023-03-12 02:20:39 +01:00
ac0f9ff5d3
feat(frontend): manage API tokens in Frontend
2023-02-21 23:07:02 +01:00
8f7eebb806
feat(api): API tokens for authentication
...
Create and managed simple API tokens for access to the API from external
tools.
2023-02-19 16:19:40 +01:00
0a9956e1ae
feat(api): add local repl console
...
Based on the REPL from NestJS 9
2022-09-27 18:14:49 +02:00
99cc06bbbc
chore(deps): upgrade to NestJS 9
...
chore(deps): update dependency nestjs-otel to v4
chore(deps): update opentelemetry-js monorepo
2022-09-27 18:14:49 +02:00
d6af6f9cba
fix(api): missing OpenAPI route tags
2022-07-12 20:34:14 +02:00
2c8e8ef23c
fix(api): remove unwanted debug log messages
2022-07-07 17:18:54 +02:00
41dfae3c50
feat(api): metrics for Spotify API http requests
...
This will help evaluate how much we are being rate limited, and on what
routes it happens most.
2022-06-29 21:30:09 +02:00
85c31705ef
refactor(api): replace deprecated toPromise calls
...
Details: https://rxjs.dev/deprecations/to-promise
2022-06-29 21:30:09 +02:00
be38c383ef
fix(api): some listens pages are not full
...
Without this option, some pages on the listens list are returning less than
requested items.
2022-06-25 16:50:48 +02:00
renovate[bot]
ef84800ce8
chore(deps): update typeorm
2022-06-25 16:26:58 +02:00
8d9e99039c
fix(api): send trace context to sentry
...
This helps associating errors in sentry with the full trace context
2022-06-23 18:26:42 +02:00
fd3fc6b159
fix(api): missing await
2022-06-19 20:35:44 +02:00
99a4593774
fix(api): improve performance of all time listens report
...
The current algorithm is CPU intensive and blocks the event loop for
multiple seconds in my deployment. This is not acceptable, as other
requests can not be answered during that time.
I do not have time to fully fix the issue here, but I did implement an
optimization for ALL_TIME reports:
Before, the all time report was generated for every timeFrame since 1970,
which iterated over the listens many hundred times. We can instead only start
the interval at the day of the first listen, and therefore skip 50+ years
of calculations.
2022-06-19 20:35:30 +02:00
6b1640b753
feat(observability): Replace Prometheus package with OpenTelemetry
2022-06-18 17:48:16 +02:00
a2ea89ff96
fix(api): broken tests after linting fix
2022-06-11 22:55:13 +02:00
84cf3391f1
chore(deps): update typescript-eslint monorepo to v5
2022-06-11 19:21:00 +02:00
ee5bd41a37
feat(api): configurable max pool connections
...
This should help with failing health checks while the crawler is running.
Quick math: 10 users, 30 songs each, each song requires at least 3
queries => 900 db queries every minute.
With the default of 10 pool connections, this blocks all available db
bandwidth for some time and causes slow UI and failing healthchecks.
2022-06-11 19:21:00 +02:00
7de5e14d4e
fix(health): remove faulty health check to spotify-auth
...
The endpoint returns a 503 OFTEN (~1x per hour), which causes unnecessary
service restarts and issues in sentry.
2022-06-11 19:21:00 +02:00
f06b93efbe
fix(health): use not-deprecated sentry context
...
The setExtra function is deprecated in favor of setContext. Also update
sentry-related packages.
2022-06-11 19:21:00 +02:00
dbf4374aeb
fix(health): send healthcheck result to sentry to improve debugging
2022-06-11 14:02:48 +02:00
a0ffe108e1
feat(api): update existing artists in MusicLibrary
2021-11-21 15:53:49 +01:00
a0c28e2324
feat: top genres report
2021-11-21 15:53:49 +01:00
Renovate Bot
19d7700696
chore(deps): update dependency passport-spotify to v2
2021-08-15 16:14:52 +02:00
Renovate Bot
a3c6da2749
chore(deps): update nest monorepo
2021-08-12 17:08:23 +02:00
e476243b85
test: create initial unit tests
2021-07-18 21:57:38 +02:00
3695fb7182
refactor(api): replace deprecated DNSHealthCheck
2021-06-22 21:06:35 +02:00
73bba4ef68
refactor(api): remove deprecated function and rename exception
2021-06-22 20:34:55 +02:00
3c6f3289f1
feat(server): save genres for artists and albums
...
This can later be used for reports
2021-05-25 19:23:42 +02:00
9b96d0fab4
chore(lint): switch to eslint
2021-05-25 18:12:42 +02:00
f56548e432
chore: run prettier format
2021-05-25 16:02:00 +02:00
51fd78f6d9
feat: add top tracks report
2021-05-22 15:10:05 +02:00
