0fceb4686b
Added missing mock data
2022-03-26 20:52:27 +01:00
70b20f0d6b
Updated mock data after conflicts
2022-03-26 20:52:27 +01:00
6a6c9359af
Fixed kurzgesagt test - They changed their description...
2022-03-26 20:52:27 +01:00
26596215fa
Refactored ``MediaCCCRecentListExtractorTest``
2022-03-26 20:52:26 +01:00
53962bfd7d
Disabled suggestion test for now
...
Has to be done once YT's backed works again...
2022-03-26 20:52:26 +01:00
66dc5e8bb8
API hardening against changes
2022-03-26 20:52:26 +01:00
5d58156cde
Update mock data
2022-03-26 20:52:25 +01:00
7598b40957
Workaround for incorrect duration for "YT shorts" videos in channels
...
As a workaround 0 is returned as duration for such videos.
See also https://github.com/TeamNewPipe/NewPipe/issues/8034
2022-03-26 20:52:24 +01:00
164e21b5af
Fixed ``MediaCCCRecentKiosk``
...
Ignore faulty data/items (with duration <= 0)
2022-03-26 20:52:23 +01:00
49dfdae5d2
Removed useless throw declaration
2022-03-26 20:51:38 +01:00
69a58fd3d1
Don't use sysout
2022-03-26 20:51:37 +01:00
639be7adda
Minimized some code
2022-03-26 20:51:37 +01:00
2bd4299563
Fixed test: Peertube - Use new comments and videos
...
Old comments/videos got deleted
2022-03-26 20:49:39 +01:00
606a386a98
Fix tests: What's peertube has less comments now
2022-03-26 20:49:39 +01:00
4d1a1c8fb8
Better test for ``MediaCCCRecentListExtractorTest``
...
* Use assertAll
* Show which item is affected
2022-03-26 20:49:38 +01:00
ba43dbaa28
Fix Tests: The channel lost abos
2022-03-26 20:49:38 +01:00
ff436e5740
Fixed test: YT mix playlist
2022-03-26 20:49:38 +01:00
9c07e8a664
Fix useage of wrong object
2022-03-26 20:17:50 +01:00
804e57004f
Fixed new checkstyle problems from dev
2022-03-26 19:46:10 +01:00
33347ac18b
Removed unused methods
...
``contentFilters`` and ``sortfilter`` are get inside the ``ListLinkHandler`` and not the ``ListLinkHandlerFactory``
``ListLinkHandlerFactory`` only passes these values through when ``fromQuery`` is called
2022-03-26 19:43:11 +01:00
ec5b54c38b
Removed unused class
2022-03-26 19:43:10 +01:00
8771af7ba5
Restored original naming
2022-03-26 19:43:09 +01:00
bdadcfa1f7
Legitimately suppress remaining checkstyle warnings
2022-03-26 19:43:08 +01:00
740a37a2de
[YouTube] Fix checkstyle issues
2022-03-26 19:42:40 +01:00
9dc17cd1ca
[Soundcloud] Fix checkstyle issues
2022-03-26 19:40:20 +01:00
9ab32cb2e7
[Peertube] Fix checkstyle issues
2022-03-26 19:40:19 +01:00
9f7e06c817
[MediaCCC] Fix checkstyle issues
2022-03-26 19:40:18 +01:00
3a94839359
[Bandcamp] Fix checkstyle issues
2022-03-26 19:40:17 +01:00
08dff33002
Use Java 8 streams in NewPipe class
2022-03-26 19:40:15 +01:00
c2446ecff0
Use Java 8 streams and deduplicate code in MediaFormat class
2022-03-26 19:40:15 +01:00
d79e20340c
Fix checkstyle issues in root package extractor/
...
Note: not all issues were fixed because MediaFormat and ServiceList use a specific formatting that makes sense for them
2022-03-26 19:40:14 +01:00
ca7c63f273
Fix remaining checkstyle issues in utils/ subpackage
2022-03-26 19:40:13 +01:00
1d5f22e41f
Fix checkstyle issues & more in JsonUtils
...
Also use Java 8 streams and extract duplicate code to getInstanceOf function
2022-03-26 19:40:13 +01:00
87d2834986
Fix checkstyle issues & more in DonationLinkHelper
...
Also add comment about the class being unused and replace the fixLink function with Utils.stringToUrl()
2022-03-26 19:40:12 +01:00
bd7b362040
Fix checkstyle issues & more in DashMpdParser
...
Also remove useless null check on ItagItem.getItag() as that function already throws an exception if there is no itag
2022-03-26 19:40:11 +01:00
8aba2b47b0
Fix checkstyle issues in subpackages with abstract classes
2022-03-26 19:40:10 +01:00
e4951a0623
Refactor code handling http headers in downloader.Request
2022-03-26 19:37:47 +01:00
37690058d2
Add checkstyle to extractor gradle project
...
With respect to NewPipe's checkstyle.xml, checkstyle is disabled for javadoc comments. There is no need for strict rules over comments here in the extractor, as sometimes javadocs are just needed to clarify a small thing and having empty/meaningless @param or @throws is useless.
2022-03-26 19:37:46 +01:00
9284569c84
Merge pull request #774 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.14
...
Bump rhino from 1.7.13 to 1.7.14
2022-03-26 17:25:55 +01:00
aa6b7272a4
Merge pull request #804 from Stypox/fix-yt-music-mix
...
[YouTube] Fix music mixes in some countries
2022-03-20 08:35:56 +01:00
09ddb6adbb
[YouTube] Add MockOnly to method testing mixes in related items
2022-03-19 10:54:38 +01:00
8201b3b90e
[YouTube] Parse any playlist (including music mixes) in related items
2022-03-19 10:48:13 +01:00
13f7900816
[YouTube] Add test for genre mix
2022-03-19 10:48:13 +01:00
279f3a20fe
[YouTube] Fix mix tests with invalid video ids
...
Replaces mix tests based on a strange mix type RDQM{videoId} (only reference I could find is https://github.com/ytdl-org/youtube-dl/issues/26228 ) and with an invalid video id of 13 characters (the first two characters were QM, but even after removing QM there still wasn't a video available at that id).
Also updates mocks.
2022-03-19 10:48:13 +01:00
d660c04838
[YouTube] Also test playlist type in playlist tests
2022-03-19 10:48:13 +01:00
401082abe4
[YouTube] Extract playlist type in playlist extractor
2022-03-19 10:48:12 +01:00
63ed06a710
[YouTube] Differentiate genre mixes from normal mixes
...
Note: genre mixes already worked, now they are just considered as such in various video id extraction and in related items
Note 2: now extracting a mix id from a *normal* youtube mix id will fail if the video id wouldn't be exactly 11 characters long
2022-03-19 10:46:31 +01:00
f19660e7d9
[YouTube] Deduplicate code extracting video id from mix id
2022-03-19 10:46:30 +01:00
8f9d5b858e
[YouTube] Remove useless comments about mixes
2022-03-19 10:44:06 +01:00
34a4484c72
[YouTube] Add test for a video with a mix in related items
2022-03-19 10:44:06 +01:00
50db871d89
[YouTube] Extract mixes from streams related items
2022-03-19 10:44:06 +01:00
638da1756c
[Mix] Create MultiInfoItemsCollector
...
It is a collector that can handle many extractor types, to be used when a list contains items of different types (e.g. search). It was renamed from InfoItemsSearchCollector so that it can now be used not just for search but for any extractor needing it. It supports, streams, channels, playlists and *mixes*.
2022-03-19 10:44:06 +01:00
53673d64c6
[Mix] Add type to playlists & playlist items, to distinguish mixes
2022-03-19 10:44:06 +01:00
d8f2031619
Merge pull request #816 from Stypox/mock-only-extension
...
Add `@MockOnly` Junit 5 extension
2022-03-19 10:40:38 +01:00
cc2e4d7104
Merge pull request #815 from litetex/fix-soundcloud-id-once-and-for-all
...
Removed hardcoded soundcloud HARDCODED_CLIENT_ID
2022-03-17 13:54:08 +01:00
c7757c0994
Apply requested changes
2022-03-16 20:14:08 +01:00
35e082248e
Fix YouTube and SoundCloud playlists tests
2022-03-16 19:40:30 +01:00
8b3f90eb7e
[YouTube] Fix extraction of series playlists and don't return the view count as the stream count for learning playlists
...
ITEM_COUNT_UNKNOWN is returned when the JSON array which contains usally the number of videos is less than 3 items.
Also apply the same type of optimizations done in other PlaylistExtractors in YoutubePlaylistExtractor.
2022-03-16 19:18:58 +01:00
58a247907e
Apply changes in all playlist extractors except YoutubePlaylistExtractor
...
Also fix some issues in the extractors, remove uneeded overrides, use the Java 8 Stream API where possible and replace usages of Utils.UTF_8 with StandardCharsets.UTF_8 in these classes.
2022-03-16 19:18:57 +01:00
fc6b45ee36
Implement some methods in PlaylistExtractor
...
This will prevent their override in each child class where the values corresponding to the methods could not be extracted.
2022-03-16 19:18:36 +01:00
73d1fd472f
Add MockOnly junit 5 test extension
2022-03-16 19:03:08 +01:00
ef71a5fa0f
static final instead of final static
2022-03-16 17:24:33 +01:00
0c37c75981
Make getDownloader static & extract getDownloaderType
2022-03-16 17:22:42 +01:00
40aa5104b1
Merge pull request #786 from XiangRongLin/throttling_resilience
...
[Youtube] Make throttling decryption more resilient to api change
2022-03-16 11:03:16 +01:00
ba56be8ef1
Removed hardcoded soundcloud id
...
It never works (long enough) so let's simply remove it...
2022-03-15 21:19:19 +01:00
e726437da3
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
...
Co-authored-by: Stypox <stypox@pm.me>
2022-03-15 17:10:05 +01:00
e7aee0ca57
Merge pull request #807 from FireMasterK/no-commentsinfo-instance
...
Remove the need for a CommentsInfo instance in CommentsInfo.getMoreItems and fix PeertubeCommentsExtractorTest.Default test
2022-03-15 15:06:56 +01:00
d806984aa8
Merge pull request #800 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.9.0
...
Bump gson from 2.8.9 to 2.9.0
2022-03-14 19:47:54 +01:00
60cc71e944
Remove the need for a CommentsInfo instance.
2022-03-03 11:48:41 +00:00
dd8687f9fe
[YouTube] Fix music mixes in some countries
2022-03-01 23:02:56 +01:00
e6d334765d
Apply requested codestyle improvements.
2022-02-26 17:12:51 +00:00
6950b362f2
Add test values for failing tests.
2022-02-24 12:55:13 +00:00
5b0ec694a6
Add requested changes.
2022-02-24 12:50:51 +00:00
d290d2e393
Move variable to super Extractor classes.
2022-02-24 12:50:51 +00:00
ab49cb6e18
Add requested changes.
2022-02-24 12:50:51 +00:00
4bd59f65f4
Update exception message.
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-02-24 12:50:51 +00:00
6f3f608ab6
Add support for extracting channel subscriber count in StreamInfo.
2022-02-24 12:50:51 +00:00
1a67ea100a
Refactored code
...
according to review
2022-02-21 19:56:12 +01:00
f79ce1f52a
Refactored YoutubeCommentsExtractor
...
* Use Java Streaming API
* Use StandardCharsets
* Prevented several NPEs/ArrayIndexOutOfBound
* Reformatted some code so that it's easier readable
2022-02-21 19:56:09 +01:00
d337e537dc
[YouTube] Add a comment clarifying why the panelIdentfier can be null
2022-02-18 20:26:19 +01:00
5c8fbeb166
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments() ( #797 )
...
Fix NullPointerException in YouTubeStreamExtractor.getStreamSegments()
Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2022-02-16 23:39:26 +01:00
89900431ab
Bump gson from 2.8.9 to 2.9.0
...
Bumps [gson](https://github.com/google/gson ) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-02-14 09:15:09 +00:00
0d6aebce25
[SoundCloud] Update clientId ( #798 )
2022-02-11 17:16:32 +01:00
a2a41e3bf2
Bump rhino from 1.7.13 to 1.7.14
...
Bumps [rhino](https://github.com/mozilla/rhino ) from 1.7.13 to 1.7.14.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2022-02-07 20:29:51 +00:00
545522b80f
[Youtube] Make throttling decryption more resilient to api change
2022-02-01 19:57:34 +01:00
047d75a4c9
Improve code style
2022-02-01 12:56:59 +01:00
50a4b026f8
Adding removal of the braces
2022-01-31 23:52:31 -08:00
a9174f6b9f
Changed the regex to account for
...
nonword characters
2022-01-31 22:29:00 -08:00
59e0aeba88
Merge pull request #768 from litetex/junit-5
...
JUnit 5
2022-01-24 16:56:16 +01:00
65df39b3e3
[YouTube] parse timestamps with >3 digits correctly ( #775 )
...
* [YouTube] parse timestamps with >3 digits correctly
Fixes https://github.com/TeamNewPipe/NewPipe/issues/7530 ; check the
issue for details.
* Remove outdated comment
2022-01-12 10:45:06 +01:00
a579ef2651
Fixed Soundcloud tests
2022-01-07 14:27:24 +01:00
a32edce750
Removed deprecation because AndroidSDK/API < 19
2022-01-07 14:15:55 +01:00
602068937d
Code cleanup
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2022-01-06 21:04:16 +01:00
dd3f3da828
Kurzgesagt mock data is outdated
2022-01-04 17:44:12 +01:00
a8ebd2cead
Update mock resources for YT Mix playlist
2022-01-04 17:39:14 +01:00
066afd8629
Restored test
2022-01-04 17:33:08 +01:00
3712a669b1
Code cleanup / improvements
...
* Deprecated Utils#UTF-8; see StandardCharsets
* Added more helpful methods to ``ExtractorAsserts``
* Use parameterized (cool new) tests
* Restore functionality of some tests + updated mockdata
* Other code cleanups + Sonarlint improvements
2022-01-04 17:28:31 +01:00
4291a90251
Fixed SoundCloud's search(for tests)
2021-12-27 21:42:43 +01:00
4995709871
Fixed SoundCloud's search(for tests)
...
Getting the initial page was not returning initial page
2021-12-27 21:18:39 +01:00
a2cbdf0991
Updated to JUnit 5
2021-12-27 21:08:08 +01:00
148ba6641d
Updated mock data
2021-12-27 17:08:14 +01:00
4d0a689cb6
Disabled dislike count in tests
2021-12-27 17:04:46 +01:00
a6961c5efa
Fixed NumberFormatException when processing likes
...
See also #7525
2021-12-27 16:54:47 +01:00
65687f3b9b
Removed YT dislike count as it no longer works
2021-12-27 16:54:46 +01:00
1e2e0029fc
[StreamExtractor] Deduplicated a ton of code by using default methods
2021-12-27 16:54:45 +01:00
15b98ffdb4
Remove old ways of getting YT dis/likes
...
* Added additional check for averageRating (in dislikes)
2021-12-27 16:54:44 +01:00
7489502cc8
Disabled ``testRelatedItems`` as it's unreliable
2021-12-11 15:23:11 +01:00
d9b2a2042a
Updated mock data for YoutubeCommentsExtractorTest
2021-12-08 22:11:45 +01:00
a3a0c1ee85
Fixed YoutubeCommentsExtractorTest
...
Old video was deleted / set to private
2021-12-08 22:11:34 +01:00
5c288eb8d7
Updated mock data for YoutubeChannelExtractorTest
2021-12-08 21:51:38 +01:00
75bb83a1b8
Fixed YoutubeChannelExtractorTest
...
Channel was restored, picked a new one
2021-12-08 21:51:15 +01:00
f276bacce5
Fixed SoundcloudStreamExtractorTest
2021-12-08 21:41:27 +01:00
7f99c1c193
Fixed YoutubeStreamExtractorLivestreamTest
2021-12-08 21:16:39 +01:00
652fdf2c36
Updated mock-data for YoutubeStreamExtractorAgeRestrictedTest
2021-12-08 21:08:25 +01:00
3c669d3cb5
Fixed YoutubeStreamExtractorAgeRestrictedTest
...
Looks like a lot of stuff (description, tags, name) got changed on the existing video.
Use a better suited video.
2021-12-08 21:08:03 +01:00
f735788a39
Fixed Soundcloud hardcoded client id
2021-12-08 21:00:53 +01:00
56c8af710e
Code refactoring
2021-11-21 18:16:31 +01:00
c4eca91be9
Fix YouTube likes + dislikes
2021-11-19 21:36:03 +01:00
5028396405
Improve names and formatting
2021-11-14 13:44:13 +01:00
68d7e4e20a
Merge pull request #742 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.9
...
Bump gson from 2.8.8 to 2.8.9
2021-11-08 19:37:09 +01:00
cb80a646d9
Update hardcoded SoundCloud client_id
...
Hah! I got here first, @TobiGr!
2021-11-04 15:28:26 +00:00
396aecef19
Make sure playlists aren't accepted by PeertubeStreamLinkHandlerFactory
2021-11-03 14:41:39 +01:00
38ad1eaac5
[PeerTube] Support /w/ short video links part 2
...
We also need to support it for comments
2021-11-03 13:54:37 +01:00
026751624f
[PeerTube] Support /w/p/ short playlist links
2021-11-03 13:35:30 +01:00
e0b8e142fc
[PeerTube] Support /a/ and /c/ short links
2021-11-03 13:26:27 +01:00
599a91c88c
[PeerTube] Support /w/ short video links
2021-11-03 11:51:58 +01:00
198d37090b
Bump gson from 2.8.8 to 2.8.9
...
Bumps [gson](https://github.com/google/gson ) from 2.8.8 to 2.8.9.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.8...gson-parent-2.8.9 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-11-01 09:12:53 +00:00
3e8e2a1532
Add support for y2u.be links
2021-10-22 22:48:18 +03:00
b425394e7a
Merge pull request #731 from FireMasterK/short-description
...
Extract Video Short Description in YouTube.
2021-10-15 23:10:00 +02:00
62313962a0
Add default implementation for getShortDescription.
2021-10-04 01:42:29 +01:00
d1add05bc1
Add requested changes.
...
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
Add final.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-10-04 01:36:51 +01:00
3ca9f51b30
Bump jsoup from 1.14.2 to 1.14.3
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.14.2 to 1.14.3.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.2...jsoup-1.14.3 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-10-02 08:20:38 +00:00
d8ec3ce370
[SoundCloud] Update client id
2021-10-01 02:51:38 +02:00
94efe86c71
Extract Video Short Description in YouTube.
...
In Trending, and Search results.
2021-09-22 16:26:16 +01:00
a9d214478d
Merge pull request #703 from FireMasterK/comment-replies
...
Add support for extracting comment replies continuation
2021-09-14 23:58:14 +02:00
6aabdc6d16
Fix for requested changes.
2021-09-12 01:15:19 +05:30
0aad09fa22
Update JavaDocs for comment replies continuation.
...
Co-authored-by: Stypox <stypox@pm.me>
2021-09-12 01:05:20 +05:30
feb09e9997
Extract Channel Avatar in related videos.
2021-09-04 01:09:40 +05:30
f0aa46b008
Merge pull request #724 from FireMasterK/streams-uploader-verified
...
Extract uploaderVerified to StreamInfo.
2021-09-03 18:18:04 +02:00
857b2f39e8
Extract uploaderVerified to StreamInfo.
2021-09-03 21:27:58 +05:30
1db463b55f
Add a default test for the Uploader's Avatar.
2021-09-03 21:05:36 +05:30
db6b3b2c29
Extract uploader's avatar in peertube.
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-09-02 16:14:45 +05:30
a0c1dcc8d8
Remove throws parsing exception.
2021-09-02 16:14:45 +05:30
0ba03c552c
Improve based on review/feedback.
2021-09-02 00:35:56 +05:30
b9fad4fcc8
Extract Uploader's Avatar on YouTube.
2021-09-02 00:19:00 +05:30
68f1fa994a
Merge pull request #721 from Stypox/yt-csv-import
...
[YouTube] Small improvements to subscription import
2021-08-31 12:02:59 +02:00
b644f1d67b
[SoundCloud] Update client id
2021-08-31 07:42:21 +02:00
a5e9eeb790
[YouTube] Small improvements to subscription import
2021-08-30 15:45:45 +02:00
9570882c73
Merge pull request #712 from magicbrothers/add-eduvid-tubus
...
Add invidious instance EduVid Tubus
2021-08-24 16:07:07 +02:00
94a29fd63f
[YouTube] csv and zip subscription import (Google Takeout)
...
csv:
Improved error messages
Exits early if it hasnt found any items in the first few lines
zip:
Now checks all CSV files instead of hard-coded paths
final qualifiers for immutable locals and parameters
Co-authored-by: litetex <40789489+litetex@users.noreply.github.com>
2021-08-24 15:42:49 +02:00
816bc7137f
Bump gson from 2.8.7 to 2.8.8
...
Bumps [gson](https://github.com/google/gson ) from 2.8.7 to 2.8.8.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.7...gson-parent-2.8.8 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-08-23 13:04:10 +02:00
72393995bb
Bump jsoup from 1.13.1 to 1.14.2 ( #716 )
...
Bumps [jsoup](https://github.com/jhy/jsoup ) from 1.13.1 to 1.14.2.
- [Release notes](https://github.com/jhy/jsoup/releases )
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES )
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.13.1...jsoup-1.14.2 )
---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-22 09:12:14 +02:00
d8179dc35a
Merge pull request #713 from TiA4f8R/regexes-yt-signatureciphers
...
[YouTube] Consolidate the regular expressions used to find the cipher decryption function
2021-08-17 18:58:27 +02:00
61acdae4b8
Fix failing PeertubeStreamExtractorTest
2021-08-17 17:06:36 +02:00
4f86b8ce43
Add verified info to ChannelInfo. ( #714 )
2021-08-17 13:08:49 +02:00
52910ce970
Keep original exception when there is an error message ( #710 )
2021-08-13 16:13:58 +02:00
f8309d546a
[YouTube] Consolidate current regexes of the cipher decryption function
...
Also add more regexes
2021-08-12 18:26:43 +02:00
0e16091ce0
Add invidious instance EduVid Tubus
2021-08-12 10:06:41 +02:00
50bba32239
Fix typos in Javadoc
2021-08-11 12:01:25 +03:00
cf9e678223
Declare common dependency versions in root gradle
2021-08-07 17:51:09 -04:00
71b9fd0076
Faster iframe api based player extraction. ( #694 )
...
* Faster iframe api based player extraction.
Uses the IFrame API to reduce the required download to less than 1/50 of the size.
* Remove debug code.
* Extract to two methods.
* Add tests for player URL extraction.
* Add assertThat for tests.
2021-08-07 12:35:48 +02:00
ee32317315
Update supported countries list for YouTube.
2021-08-05 23:48:38 +05:30
e8e9e6cd00
Fix comment reply continuations.
2021-08-04 22:14:04 +05:30
666b45958e
Fix test and update mocks.
2021-08-04 21:10:02 +05:30
f6d054e5da
Add test for comments.
2021-08-04 21:03:00 +05:30
ed84658055
Get it working.
2021-08-04 18:35:14 +05:30
ea7e6526fd
Collect replies/page to CommentsInfoItem.
2021-08-04 18:09:42 +05:30
4385cc1b7c
Add comment replies.
2021-08-04 17:56:04 +05:30
5b38b3ae97
Merge branch 'master' into v0.21.8
2021-08-03 20:38:07 +02:00
79cc5c8d12
Fix tests of YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest
...
Without removing RunWith and SuiteClasses annotations (and the corresponding imports) in YoutubePlaylistExtractorTest and YoutubeMixPlaylistExtractorTest, some mocks cannot be generated, so the CI fails because of the missing mocks. Mocks of workings tests have been also updated.
2021-08-01 17:28:01 +02:00
b74a39c176
Reformat some code and don't use the clickTrackingParams in continuations of YouTube Music search results
...
The clickTrackingParams of YouTube Music search results are not needed to get continuations. This commit removes their use, which may improve privacy.
2021-08-01 17:25:59 +02:00
f8197da09e
Update YouTube comments and YouTube playlist continuations mocks
2021-08-01 12:39:05 +02:00
f3e4c9d689
Use the youtubei API for YouTube comments
...
Migrate YouTube comments to the desktop version by using the `next` endpoint of the InnerTube internal API.
With the desktop version, we are able to get the exact like count of YouTube comments (by parsing the accessibility data) (the current extraction is used as a fallback). We are also now able to get if the uploader of the comment is verified or not.
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:05 +02:00
286d839a3b
Update mocks (except mocks of YoutubeCommentsExtractorTest)
2021-08-01 12:39:04 +02:00
693f654f02
Ignore a broken test of YoutubeMixPlaylistExtractorTest
2021-08-01 12:39:03 +02:00
d55109258d
Fix YoutubeStreamExtractorAgeRestrictedTest
2021-08-01 12:39:03 +02:00
2eeb0a3403
Rebase + some code improvements + fix extraction of age-restricted videos + update clients version
...
Here is now the requests which will be made by the `onFetchPage` method of `YoutubeStreamExtractor`:
- the desktop API is fetched.
If there is no streaming data, the desktop player API with the embed client screen will be fetched (and also the player code), then the Android mobile API.
- if there is no streaming data, a `ContentNotAvailableException` will be thrown by using the message provided in playability status
If the video is age restricted, a request to the next endpoint of the desktop player with the embed client screen will be sent.
Otherwise, the next endpoint will be fetched normally, if the content is available.
If the video is not age-restricted, a request to the player endpoint of the Android mobile API will be made.
We can get more streams by using the Android mobile API but some streams may be not available on this API, so the streaming data of the Android mobile API will be first used to get itags and then the streaming data of the desktop internal API will be used.
If the parsing of the Android mobile API went wrong, only the streams of the desktop API will be used.
Other code changes:
- `prepareJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareDesktopJsonBuilder`
- `prepareMobileJsonBuilder` in `YoutubeParsingHelper` was renamed to `prepareAndroidMobileJsonBuilder`
- two new methods in `YoutubeParsingHelper` were added: `prepareDesktopEmbedVideoJsonBuilder` and `prepareAndroidMobileEmbedVideoJsonBuilder`
- `createPlayerBodyWithSts` is now public and was moved to `YoutubeParsingHelper`
- a new method in `YoutubeJavaScriptExtractor` was added: `resetJavaScriptCode`, which was needed for the method `resetDebofuscationCode` of `YoutubeStreamExtractor`
- `areHardcodedClientVersionAndKeyValid` in `YoutubeParsingHelper` returns now a `boolean` instead of an `Optional<Boolean>`
- the `fetchVideoInfoPage` method of `YoutubeStreamExtractor` was removed because YouTube returns now 404 for every client with the `get_video_info` page
- some unused objects and some warnings in `YoutubeStreamExtractor` were removed and fixed
Co-authored-by: TiA4f8R <74829229+TiA4f8R@users.noreply.github.com>
2021-08-01 12:39:03 +02:00
7753556e66
Adress the last requested changes + update YoutubeCommentsExtractor mocks
2021-08-01 12:39:03 +02:00
2320aecb7c
Update mocks
2021-08-01 12:39:02 +02:00
629dcd63b2
Ignore some broken tests
2021-08-01 12:39:01 +02:00
3adac6a150
Workaround for rate limits: always use the Android mobile API
2021-08-01 12:39:01 +02:00
8aa60d7e8f
Update clients version
2021-08-01 12:39:01 +02:00
27bd797f1e
Update mocks
2021-08-01 12:39:01 +02:00
609919db59
Adress again reviews, fix some rebase issues
2021-08-01 12:39:00 +02:00
4299d806a2
Adress changes
2021-08-01 12:38:59 +02:00
1a6b8da438
Annotate YoutubeParsingHelper methods with Nonnull when needed
2021-08-01 12:38:59 +02:00
d8177b57f6
Loop in all formats to check if the stream has URLs protected by signatureCiphers
2021-08-01 12:38:59 +02:00
a6a2c6eb80
Revert the use of Collections.singletonList instead of Arrays.asList in addCookieHeader of YoutubeParsingHelper
2021-08-01 12:38:59 +02:00
81013e5a8e
Add a static method to reset the YouTube deobfuscation code and use it in tests
...
This method is needed for YouTube stream tests, because when all YouTube tests are ran, the signatureTimestamp is known (the sts string) so a different body than the body present in the mocks is send by the extractor instance.
As a result, running all YouTube stream tests with the MockDownloader (like the CI does) will fail if this method is not called before fetching the page of a test.
2021-08-01 12:38:58 +02:00
cc798523cd
Adress requested changes in other classes
2021-08-01 12:38:58 +02:00
accd5ddef3
Adress requested changes in YoutubeStreamExtractor
2021-08-01 12:38:58 +02:00
632772d17f
Adress requested changes in YoutubeParsingHelper
2021-08-01 12:38:58 +02:00
657f165771
Update client version and mocks
2021-08-01 12:38:44 +02:00
8c1c7281b0
Real fix of the fetch of the JS player at each the signatureTimestamp is required
...
The strings playerJsUrl, sts and playerCode are now static in order to don't fetch again the JavaScript player at each time the signatureTimestamp is needed.
2021-08-01 12:38:43 +02:00
e97a685989
Format the YoutubeMusicSearchExtractor class to be in the 100 characters per line limit
2021-08-01 12:38:43 +02:00
34a9ccb0fd
Adress requested changes
2021-08-01 12:38:42 +02:00
54d4551ca6
Adress requested changes in YoutubeParsingHelper and update mobile client version
2021-08-01 12:38:42 +02:00
a59c2a3577
Catch every exception instead of only IOException and ExtractionException and add a Javadoc
...
Catch every exception instead of only IOException and ExtractionException.
Add JavaDoc for fetchAndroidMobileJsonPlayer method of YoutubeStreamExtractor
2021-08-01 12:38:42 +02:00
6921e80ded
Try again to don't fetch at each time the JavaScript player
2021-08-01 12:38:42 +02:00
70927ddade
Update client version and mocks
2021-08-01 12:38:40 +02:00
318bc46a8c
Readd the deleted code of views because watching count of livestreams was broken
...
The number shown was the total number of views that a livestream has. In order to fix this bug, the previous code is readded.
2021-08-01 12:38:39 +02:00
947baec805
Fetch again the desktop player JSON only if the content is protected by signatureCiphers
2021-08-01 12:38:39 +02:00
7474049fd1
Update mocks
2021-08-01 12:38:39 +02:00
c32bc6e534
Try to don't fetch again the first page when requesting a channel continuation
...
Try to don't fetch again the first page of a YouTube channel when requesting a continuation of it by trying to store the channel name and the channel id into the next page using the ids field of the Page class.
2021-08-01 12:38:38 +02:00
b52732a1c3
Fix the fetch of the playerCode at each time getStsFromPlayerJs is called
...
storePlayerJs was called even if sts was not empty in the getStsFromPlayerJs method. This commit fixes it.
2021-08-01 12:38:37 +02:00
ae5abc0c5d
Fix a typo in YoutubeChannelExtractor
2021-08-01 12:38:37 +02:00
0f9e9b8b4b
Use the youtubei API for YouTube mixes + update the corresponding test + do some improvements
...
Use the youtubei API for YouTube mixes. The corresponding has been updated because the new API breaks the tests of YoutubeMixPlaylistExtractorTest.
Remove some deprecated code (the old search code with the pbj JSON) and do some other improvements.
2021-08-01 12:38:37 +02:00
14569c4aa9
Readd the return of itags 134 and 136
2021-08-01 12:38:37 +02:00
3017dde67e
Fix some typos
2021-08-01 12:38:36 +02:00
013b902535
Use the Android mobile API when there are OTF streams or the content is protected by signatureCiphers
...
Use the Android mobile API to get the itag 22 (720p with audio), removed when the content is protected by signatureCiphers.
Also use this API when they are OTF streams, to get the itag 17 and 36, low 3GPP quality streams but also the itag 139.
Update the web client version.
2021-08-01 12:38:36 +02:00
e7d589edbf
Use the youtubei API for YouTube videos + update client version
...
Update the hardcoded client version to 2.20210520.09.00
Use the player and next endpoints of the Innertube API for YouTube videos
2021-08-01 12:38:36 +02:00
f73c923f60
Don't use the youtubei.googleapis.com but the websites domains + update client version of the desktop internal API
...
Use again www.youtube.com and music.youtube.com domains instead of youtubei.googleapis.com domain because it spoofs more a web client of YouTube or YouTube Music and may reduce Google's detection of NewPipe Extractor users.
2021-08-01 12:38:34 +02:00
effcdaa4f2
Update mocks, reenable a test and fix a test
...
This commit updates mocks and reenables the test invalidId of the NotAvailable class of the YoutubePlaylistExtractorTest class beacuse with the youtubei/innertube API, it returns "Not found" and doesn't redirect to the YouTube homepage.
The expectedMetaInfo test of the MetaInfoTest class of the YoutubeSearchExtractorTest class was broken because YouTube removes the vaccine progress link of the WHO from the meta info so this commit removes it and the test is now passing.
2021-08-01 12:38:04 +02:00
4d682834c3
Fix localization and update client version
2021-08-01 12:38:03 +02:00
f46cfb0f26
Adress reviews and do some improvements
...
Adress changes requested in reviews.
Do some improvements, remove unused imports and format some code to be in the 100 characters line limit.
2021-08-01 12:38:03 +02:00
e075dd5a63
Update client version, fix some tests, update mocks and do some improvements
...
Add the origin and the referer headers with the https://www.youtube.com value for YouTube JSON POST requests.
Don't add the consent cookie header for the requests which use the youtubei/innertube API because it's uneeded.
Fix some tests and update YouTube mocks
2021-08-01 12:38:02 +02:00
b49ae547a3
Do some improvements to YoutubeStreamExtractor
...
Get the real name of the uploader (for autogenerated channels and music artist channels), like before the migration to the JSON pbj.
Do some other improvements, especially reformatting some code to be in the 100 characters line limit and use final where possible.
2021-08-01 12:38:01 +02:00
58ce9b04a1
Fix channel extraction when channel URL is youtube.com/c/username
2021-08-01 12:38:01 +02:00
991b2c7d73
Use lightweight requests when getting and checking YouTube API key and client version
2021-08-01 12:38:01 +02:00
9ab9c66ddf
Use the youtubei.googleapis.com domain for YouTube Music searches + change a check + update client version and mocks
...
Change the domain from music.youtube.com to youtubei.googleapis.com.
Use a lightweight request to check if the hardcoded YouTubeMusic keys are valid. Increase the length of the response to 500 because if the key is invalid, the length of the response returned is higher than 250 and the response when the key is valid is higher than 1500.
Format the YoutubeMusicSearchExtractor file.
Update YouTube web client version and mocks
2021-08-01 12:37:56 +02:00
77c031a88a
Use the youtubei API for YouTube trends
2021-08-01 12:36:28 +02:00
f461224b2b
Use the youtubei API for YouTube searches + update mocks
...
Add getSearchParameter, a new method in YoutubeSearchQueryHandlerFactory class which returns the params field for a search, or an empty string if there is no one.
Update mocks of YoutubeSearchExtractorTest.
2021-08-01 12:36:28 +02:00
a12c69da7d
Use the youtubei API for YouTube channels
2021-08-01 12:36:27 +02:00
5794eb2350
Use the youtubei API for YouTube playlists
2021-08-01 12:36:12 +02:00
c97a19d719
Merge pull request #696 from XiangRongLin/decrypt_pattern
...
Expand regex to match n param decrypt function
2021-08-01 10:24:22 +02:00
37df225556
Remove length check from StringUtils.matchToClosingParenthesis
2021-07-31 16:05:24 +02:00
f4aad8b014
Fix tests again.
2021-07-31 14:40:51 +05:30
88c11db4c5
Fix isCorrectedSearch test.
2021-07-31 02:18:18 +05:30
fd19c53f65
Fix search suggestions test.
2021-07-31 02:08:50 +05:30
a685941bba
Fix YouTube music search.
2021-07-31 01:33:38 +05:30
852a65ff18
Add tests for StringUtils
2021-07-30 19:56:57 +02:00
48d897e6ad
Add final and adjust utils class name
2021-07-29 21:33:45 +02:00
60794aea31
[YouTube] Add parenthesis matching as way to parse decrypt function
...
From @Stypox
2021-07-29 21:23:00 +02:00
2967d1ae6a
[YouTube] Compile YoutubeThrottlingDecrypter pattern statically
2021-07-29 20:27:29 +02:00
1c78976900
[YouTube] Expand regex to match n param decrypt function
...
Temporary solution
2021-07-29 20:26:36 +02:00
b62fe7141e
Fix JDoc 2
2021-07-29 12:34:04 +02:00
dbc4e01c17
Fix JDoc
2021-07-29 12:31:33 +02:00
8bfcb0ad59
[SoundCloud] Fix hardcoded client id
2021-07-28 20:33:21 +02:00
027dc65434
pull request #683 from XiangRongLin/yt_throttling
...
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 20:32:38 +02:00
14c179f343
Merge pull request #692 from TeamNewPipe/fix/sc/id
...
[SoundCloud] Fix hardcoded client id
2021-07-28 18:28:07 +02:00
394c02ad06
Merge pull request #683 from XiangRongLin/yt_throttling
...
[YouTube] Fix buffering by decoding n parameter of stream urls
2021-07-28 18:01:57 +02:00
785ff4aa32
[SoundCloud] Fix hardcoded client id
2021-07-28 17:02:56 +02:00
79f2d74b04
Fix test
...
The decryption code changes over time. Only check whether the n parameter was changed in the URL
2021-07-28 16:45:31 +02:00
d70adfdb8f
Add methods for cache control to YoutubeThrottlingDecrypter.
2021-07-27 20:32:22 +02:00
d13f531b6f
Use YoutubeThrottlingDecrypter also in getAudioStreams and getVideoOnlyStreams methods of YoutubeStreamExtractor
...
Without this commit, the n param is only decrypted for streams extracted in getVideoStreams (so only for streams in the formats object of the player response).
2021-07-27 17:28:32 +02:00
32055147e0
Do some code improvements
...
Use final where possible, annotate some methods and parameters as Nonnull and format new code to be in the 100 characters limit per line.
2021-07-27 17:24:27 +02:00
1c30a2725e
Cache nParams to prevent executing the JavaScript function for the same nParam multiple times.
...
Closes #689
2021-07-22 19:40:12 +02:00
3a3d1d7f2b
Make YoutubeJavaScriptExtractor and JavaScript methods static
...
Also address review and rewrite some comments
2021-07-20 20:48:11 +02:00
a683c8d278
Delete duplicated code to load youtube javascript file
2021-07-17 19:14:57 +02:00
a02ee2e952
Rewrite youtube throttling solution and add tests
2021-07-17 19:10:09 +02:00
0c12b396e5
Review changes
2021-07-17 16:41:12 +02:00
ecf4232ce3
Using ``Collections.emptyList()``
2021-07-17 16:41:12 +02:00
0f9ed020a4
Frameset has to implement Serializable or NewPipe fails
...
java.lang.RuntimeException: Parcelable encountered IOException writing serializable object (name = org.schabi.newpipe.extractor.stream.StreamInfo)
...
Caused by: java.io.NotSerializableException: org.schabi.newpipe.extractor.stream.Frameset
...
2021-07-17 16:41:11 +02:00
17ccaf4b87
Update YoutubeStreamExtractor.java
...
To fix ``java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.String[] java.lang.String.split(java.lang.String)' on a null object reference``
2021-07-17 16:41:10 +02:00
4e41e172ea
Update Frameset.java
...
Reformatted code
2021-07-17 16:41:10 +02:00
09b670d745
Renamed frames to previewFrames (so it's better understandable)
2021-07-17 16:41:09 +02:00
b086655d54
Reverted useless change
2021-07-17 16:41:08 +02:00
93b0c5b15f
Added seekbar data
2021-07-17 16:41:08 +02:00
6956b72af7
Fix YoutubeThrottlingDecoder.replaceNParam
...
Previously it replaced the parameter itself not the value of the parameter.
2021-07-17 10:05:14 +02:00
a189f685dc
cache player js code
2021-07-16 19:20:40 +02:00
fcdb9bdbeb
add url to sout
2021-07-15 20:05:26 +02:00
80cf8b3acd
Extract separate YoutubeThrottlingDecoder
2021-07-15 20:05:25 +02:00
a86a30103f
[Youtube] bare bones version to solve throttling
...
Done by transforming the parameter "n" from videoplayback urls
https://github.com/ytdl-org/youtube-dl/issues/29326#issuecomment-865985377
2021-07-15 20:05:25 +02:00
c38a06e8dc
Merge pull request #661 from B0pol/peertube-livestreams
...
[PeerTube] Support livestreams
2021-07-13 15:48:13 +02:00
fdebf3c6cd
Added javadoc warnings
2021-07-07 20:41:59 +02:00
6860543b07
Added more doc
2021-07-06 21:16:31 +02:00
a59fead0d7
Fixed typos
2021-07-06 21:16:30 +02:00
688a1c316b
Fixed Exception when YT comments are disabled and added ``commentsDisabled`` field
...
* Fixed code: Added missing finals (according to NewPipes Checkstyle guide)
* Fixed ``findValue`` method in ``YoutubeCommentsExtractor``
2021-07-06 21:16:29 +02:00
ca33f4f60b
Improved code format (final) and added ``isCommentsDisabled`` method
2021-07-06 21:16:28 +02:00
dbdc962a07
Update mocks
2021-07-03 13:30:26 +02:00
9ca6dc26ed
Fix invalid CONSENT cookie value
2021-07-03 13:27:40 +02:00
b9549ace25
[Youtube] Fix lofi girl test and ignore mix with id test
2021-06-29 21:12:20 +02:00
5982431ffb
Remove IPv4 addresses from response during mock recording
2021-06-29 21:03:33 +02:00
7d7cc087e2
[PeerTube] Support livestreams
2021-06-24 11:02:30 +02:00
3966178979
Merge pull request #665 from B0pol/agerestricted
...
[youtube] Fix extraction of age restricted videos
2021-06-23 21:07:14 +02:00
b475f09ba7
don't do useless concatenation
2021-06-23 20:33:32 +02:00
b96b2a6eba
Update invidious instances
2021-06-23 14:39:10 +02:00
c5e8bd368d
Update Mocks
2021-06-23 12:21:57 +02:00
e9a992b0a9
[youtube] Fix extraction of age restricted videos
2021-06-23 12:06:02 +02:00
ad403ce73e
Bump spotbugs-annotations from 4.0.2 to 4.2.3
...
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs ) from 4.0.2 to 4.2.3.
- [Release notes](https://github.com/spotbugs/spotbugs/releases )
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md )
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.0.2...4.2.3 )
---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-23 09:57:42 +00:00
65fd8740aa
Merge pull request #658 from TeamNewPipe/revert-651-fix/streamType
...
[YouTube] Fix getting stream type of live streams
2021-06-23 11:33:24 +02:00
361f142621
Use badge style instead of label
2021-06-23 11:25:59 +02:00
58f109ddec
adress stypox review
2021-06-23 11:15:40 +02:00
fe29c78fe1
Merge pull request #656 from TeamNewPipe/dependabot/gradle/org.mozilla-rhino-1.7.13
...
Bump rhino from 1.7.12 to 1.7.13
2021-06-20 00:46:42 +02:00
e9412fdadc
Merge pull request #654 from TeamNewPipe/dependabot/gradle/com.google.code.gson-gson-2.8.7
...
Bump gson from 2.8.6 to 2.8.7
2021-06-20 00:35:24 +02:00
1f9a99c931
[SoundCloud] Update HARDCODED_CLIENT_ID
2021-06-18 15:30:54 +02:00
c998012c28
Revert "[YouTube] Fix getting stream type of live streams"
2021-06-18 09:20:32 +02:00
38d0339753
Bump rhino from 1.7.12 to 1.7.13
...
Bumps [rhino](https://github.com/mozilla/rhino ) from 1.7.12 to 1.7.13.
- [Release notes](https://github.com/mozilla/rhino/releases )
- [Changelog](https://github.com/mozilla/rhino/blob/master/RELEASE-NOTES.md )
- [Commits](https://github.com/mozilla/rhino/commits )
---
updated-dependencies:
- dependency-name: org.mozilla:rhino
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:07 +00:00
5a46878b09
Bump gson from 2.8.6 to 2.8.7
...
Bumps [gson](https://github.com/google/gson ) from 2.8.6 to 2.8.7.
- [Release notes](https://github.com/google/gson/releases )
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md )
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.6...gson-parent-2.8.7 )
---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2021-06-17 21:49:01 +00:00
6762fe3357
[YouTube] Fix getting stream type of live streams
2021-06-10 16:51:00 +02:00
f7f727d19c
Add constant NO_LIKE_COUNT to CommentsInfoItem
2021-06-08 10:28:44 +02:00
b70c0f93c7
Add streamPosition for comments
...
SoundCloud is the only service which supports adding comments at a specific timestamp in the stream.
2021-06-08 10:27:52 +02:00
d4186d100b
Merge pull request #580 from TeamNewPipe/accountTerminated
...
Add AccountTerminatedException for terminated channels
2021-06-08 09:55:58 +02:00
89e0514d8b
Fix Javadoc generation fix
2021-06-08 09:30:05 +02:00
80d3052033
Fix Javadoc generation
2021-06-08 09:26:01 +02:00
fa444c8298
Remove assertations of exception messages
2021-06-06 11:38:43 +02:00
2158ca4060
Remove two unneded warning suppressions
2021-06-05 14:15:48 +02:00
13f192704d
Add utility method getService() to Info object
2021-06-05 14:14:19 +02:00
0ad51e76fb
Merge pull request #638 from TeamNewPipe/spelling
...
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-06-04 18:23:31 +02:00
faa503c7dd
Fix PeertubeAccountExtractorTest
2021-06-03 19:54:42 +02:00
ac31f3a883
Specify UTF-8 file encoding in RecordingDownloader and MockDownloader
...
On Windows, mocks are recorded and read with the Cp1252 encoding so it breaks the mocks on non ASCII characters for Linux OS (and so the CI).
The project is in Java 8, so we can't use FileReader(File, Charset) and FileReader(File, Charset) because these methods require Java 11. Instead of changing the Java version of the extractor, use OutputStreamWriter and FileOutputStream instead of FileWriter and InputStreamReader and FileInputStream instead of FileReader.
2021-06-03 18:38:25 +02:00
ff11c2df2a
Merge pull request #628 from litetex/fix-broken-yt-liked-comments
...
Fix broken yt likes in comments
2021-05-29 11:04:11 +02:00
1ff56a85df
Update junit from 4.13.1 to 4.13.2
...
https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.2.md
2021-05-28 15:09:26 +02:00
858e007804
Improve exception message thrown when not accepting an URL in LinkHandlerFactory#fromUrl(String url, String baseUrl)
2021-05-28 14:59:44 +02:00
519bba70b3
Merge pull request #630 from XiangRongLin/fix_tests
...
Fix MetaInfoTest.expectedMetaInfo
2021-05-28 14:16:53 +02:00
289b84e68b
Removed unused code
2021-05-27 21:27:44 +02:00
e81b0e2885
Tweaked code
2021-05-27 19:48:31 +02:00
8c96545e57
Merge branch 'TeamNewPipe:dev' into fix-broken-yt-liked-comments
2021-05-24 18:06:45 +02:00
12fb18c310
Updated mock data (only the affected)
2021-05-24 18:05:26 +02:00
8e08a2aac7
Improved docs
2021-05-24 18:05:26 +02:00
020acfed71
Fixed tests and added new ones
2021-05-24 18:05:26 +02:00
b310922fc0
Better exception message
2021-05-24 18:05:25 +02:00
86bb9efb5d
Added likeCount back to tests
2021-05-24 18:05:25 +02:00
46654f5fca
Reformatted code
2021-05-24 18:05:25 +02:00
2174685c5c
Reimplemented likeCount
2021-05-24 18:05:25 +02:00
01cfb55505
Update YoutubeCommentsInfoItemExtractor.java
2021-05-24 18:05:25 +02:00
bedcd87abb
Fixed occurrences where voteCount is not set (<=0 likes)
2021-05-24 18:05:25 +02:00
545c0a6f42
Fixed comment
2021-05-24 18:05:24 +02:00
a00fdcbd3d
Format the indentation of the definition of the apiUrl string in SoundcloudParsingHelper.resolveFor method
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-05-23 20:27:45 +02:00
6b607eb38d
Update the hardcoded client_id
2021-05-23 18:08:44 +02:00
4552ea9c9f
Use the SOUNDCLOUD_API_V2_URL constant in all the SoundCloud package
2021-05-23 18:08:44 +02:00
86308d0603
Use a lightweight request to check if the client_id is valid
...
Request the api-v2 host with the client_id instead of checking if the streams of a SoundCloud track are not empty: if it is valid, the API returns 404, otherwise it should return 401.
2021-05-23 18:08:43 +02:00
c5c190500c
Format the SoundCloud package of the extractor and fix some warnings
...
Use final where possible in the package and format code to be in the 100 caracters per line limit.
Fix some warnings generated by Android Studio and do some code improvements
2021-05-23 18:08:42 +02:00
636e27333b
Merge pull request #603 from TeamNewPipe/peertube_hls
...
Add PeerTube HLS streams
2021-05-23 11:49:40 +02:00
172394cd52
Update mocks
2021-05-22 20:33:20 +02:00
18c8cac40c
Fix MetaInfoTest.expectedMetaInfo
...
Expected meta changed
2021-05-21 21:00:08 +02:00
e8b5364745
Update YoutubeStreamExtractor.java
2021-05-21 16:22:24 +03:00
c3b837fe3b
Deduped code; Using default values...
2021-05-20 20:22:40 +02:00
b934c7ccbb
Removed unused import
2021-05-20 20:11:35 +02:00
10cf081145
Removed likeCount and added textualVoteCount
2021-05-20 20:10:14 +02:00
1acc53b8db
[PeerTube] Add support for HLS streams
2021-05-15 12:39:23 +02:00
3a3ade20f4
Merge pull request #620 from TeamNewPipe/bandcamp-comments
...
Add Bandcamp comments (first page)
2021-05-02 13:43:53 +02:00
85a4270bd9
[Bandcamp] Fix comments test
2021-05-02 13:07:24 +02:00
ec50570053
Add mocks
2021-05-01 17:50:19 +02:00
f4404b5cc8
Add tests for more reasons
2021-05-01 17:50:19 +02:00
fc998589dc
Detect channels which have been terminated due to copyright infringement
2021-05-01 17:50:19 +02:00
bb3861ddce
Add AccountTerminatedException for better error handling of terminated channels
2021-05-01 17:50:19 +02:00
284362fc15
Merge pull request #608 from TeamNewPipe/youtube-fix-segment-test
...
[YouTube] fix tests
2021-05-01 17:24:41 +02:00
9c12dc5609
[SoundCloud] Fix SoundCloud ID extraction
...
resolveIdWithEmbedPlayer() does not work anymore because the JSON data has been extracted to an API call. For this reason, replace resolveIdWithEmbedPlayer() with resolveIdWithWidgetApi)( which performs the API call.
2021-04-26 17:58:34 +02:00
9dc7a1d5ee
[Bandcamp] Add comments (first page)
2021-04-26 17:25:04 +02:00
8f023c1ec7
[SoundCloud] Update client id
2021-04-26 15:14:11 +02:00
7f202db8b1
Fix Google URL detection ( #617 )
2021-04-26 11:49:47 +02:00
6db4bea8ca
Rework Bandcamp comments ( #613 )
2021-04-13 21:10:59 +02:00
3671876721
Fix bandcamp radio stream extractor
2021-04-13 19:30:11 +02:00
90b5c00599
Extract related items #593
2021-04-13 17:56:19 +02:00
1fe645704f
[YouTube] FIx testMoreRelatedItems in RandomQueryNoMorePages
2021-04-11 23:21:35 +02:00
9a2b814d8f
Update mocks
2021-04-11 22:59:01 +02:00
50005ce937
[YouTube] Replace unavailable YouTube Stream Segments test
...
The stream is unavailable in many countries.
2021-04-11 22:58:48 +02:00
318cec7625
[Bandcamp] Add Radio stream segments
2021-04-11 21:19:44 +02:00
55e0f8e725
Merge branch 'master' into dev
2021-04-10 11:33:34 +02:00
33173eb3e6
Make CONSENT COOKIE constants private
2021-04-09 11:59:17 +02:00
b94b316558
Update mocks
2021-04-09 11:59:17 +02:00
4b8d4a84b6
Set the NumberGenerator for th remaining tests
2021-04-09 11:59:17 +02:00
db7ad9c27d
Update Mocks
2021-04-08 16:37:29 +02:00
4833df60e5
Add method to inject Random into YoutubeParsingHelper and use in tests
2021-04-08 16:36:55 +02:00
682ec27737
Update mocks
2021-04-08 14:22:31 +02:00
080d1e318d
Code improvement
2021-04-07 14:11:47 +02:00
4ee270fe01
Fix LiveStream Test
2021-04-07 14:11:41 +02:00
883f16e0ad
[YouTube] Set CONSENT cookie
2021-04-07 14:11:37 +02:00
aee3838ed5
Ignore unreliable SoundcloudGoPlusTrack.testRelatedItems
2021-04-06 18:26:45 +02:00
7b06c696e2
Use Page.body for YoutubeChannelExtractor
2021-04-06 18:14:38 +02:00
eda1b6e199
Use Page.body for YoutubePlaylistExtractor
2021-04-06 18:14:38 +02:00
3af26a2821
Add body field to Page
2021-04-04 11:39:58 +02:00
7b36469807
Fix SoundcloudGoPlusTrack
...
https://soundcloud.com/martinsolveig/places#t=0 has related streams
2021-04-04 09:39:26 +02:00
3d70b6d46a
Fix a test and ignore rest
2021-04-03 11:03:10 +02:00
1925dcf4dc
Merge pull request #597 from fynngodau/related-items
...
Rename getRelatedStreams to getRelatedItems and change return type
2021-04-02 19:54:22 +02:00
c877712647
Add deprecated old method calls to avoid breaking API
2021-04-01 22:32:36 +02:00
14f6f1b7c3
Generify related streams calls and rename method
2021-03-31 20:24:40 +02:00
1498e1905e
Reduce number of type conversions in YouTubeSearchExtractor.collectStreamsFrom by introducing new variable
2021-03-31 13:30:38 +02:00
7ea2cd73d3
Merge pull request #589 from fynngodau/bandcamp-more-recent-pages
...
Load more featured pages and fix featured cover arts not loading
2021-03-31 12:15:12 +02:00
705f6c6e33
Apply review
2021-03-31 11:02:30 +02:00
657b00ca11
Smaller code improvements
2021-03-30 10:10:53 +02:00
8fbd1a2bcf
FIx potential NullPointerException
2021-03-30 10:07:20 +02:00
f2e7c7ebd6
Replace Vector with ArrayList for performance improvements
...
I could not figure out, why these two methods are necessary.
2021-03-30 10:07:20 +02:00
48fd8ca7b3
Remove redundant type declaration
2021-03-30 10:07:20 +02:00
22bd16fca3
Make vars private, final and static
2021-03-30 10:07:20 +02:00
6a0f6e846a
Fix possible bug when language.id field is not present and this the languageCode an empty String.
2021-03-30 10:07:20 +02:00
b9282bbe94
Remove checks which are always true
2021-03-30 10:07:20 +02:00
46eab1ec17
Remove declarations of exceptions which are not thrown from method signatures
2021-03-30 10:07:20 +02:00
c3e23559d7
Remove unused imports
2021-03-30 10:07:20 +02:00
e4c40ae6d1
Use List.isEmpty()
2021-03-30 10:07:20 +02:00
d663be5a78
[Bandcamp] Fix featured albums not showing thumbnails
2021-03-28 13:12:22 +02:00
dbcf61c6f7
[Bandcamp] Load more featured pages
2021-03-28 12:53:05 +02:00
b4dee6d08f
Merge pull request #306 from B0pol/metadata
...
Extract metadata for youtube, soundcloud & mediaccc
2021-03-27 08:45:47 +01:00
070a40e181
[YouTube] Mix Playlist - Fix getting name
2021-03-25 21:47:16 +01:00
b029779217
[media.cc.de] Disable unreliable tests for now
2021-03-25 19:44:44 +01:00
bbee15474d
[PeerTube] Fix playlist extraction
2021-03-25 19:19:19 +01:00
ae283314da
Add a comment explaining subscribers count calculation
2021-03-25 17:54:46 +01:00
abb790f465
Remove println in getSubscriberCount from PeertubeAccountExtractor class
2021-03-25 17:49:22 +01:00
772de53a66
Increase the number of subscribers in PeertubeAccountExtractorTest
2021-03-25 17:49:20 +01:00
efe2b964f0
Extract real subscribers of a Peertube account
...
Apply fix provided in the PR and move all accounts/ strings to a constant, ACCOUNTS.
2021-03-25 17:47:44 +01:00
12835bfae1
Merge pull request #551 from FireMasterK/ff-privacy
...
Change UA to privacy.resistFingerprinting.
2021-03-24 17:58:34 +01:00
cae6cea551
Merge pull request #582 from Stypox/fix-trending
...
[YouTube] Fix trending getName()
2021-03-24 10:49:46 +01:00
ce7cbbc9a0
[YouTube] Fix trending getName()
2021-03-24 09:14:09 +01:00
9009f1e277
Merge pull request #517 from XiangRongLin/flaky_test
...
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-03-20 18:04:14 +01:00
152221c7fb
ignore a test with encoding problems
2021-03-19 10:14:43 +01:00
8806fb4e6b
address reviews
2021-03-19 10:14:43 +01:00
c47cc54908
Extract metadata for YouTube, SoundCloud & MediaCCC
2021-03-19 10:14:43 +01:00
379d7312fa
Don't use a regular expression to find the last segment URL and do code improvements
...
Apply suggestions provided in the PR and remove a redundant import.
2021-03-14 17:54:22 +01:00
0e3e420a25
Fix tests
2021-03-14 17:54:21 +01:00
d61d9d116d
Refactor getAudioStreams method of SoundcloudStreamExtractor
...
Split the method into private methods, in order to have a better reading.
2021-03-14 17:54:20 +01:00
a7b15b51e6
Change t to transcodingJsonObject in SoundcloudStreamExtractor
2021-03-14 17:54:19 +01:00
0438828e36
Add a test for the number of audioStreams in CreativeCommonsPlaysWellWithOthers test
...
It should be only two audio streams for track "Plays Well with Others, Ep 2: What Do an Army of Ants and an Online Encyclopedia Have in Common?" by Creative Commons (https://soundcloud.com/wearecc/plays-well-with-others-ep-2-what-do-an-army-of-ants-and-an-online-encyclopedia-have-in-common ):
- one which is a progressive stream, in MP3 format with a bitrate of 128 kbps
- one which is an HLS stream, in OPUS format with a bitrate of 64 kbps.
2021-03-14 17:54:18 +01:00
3bd08a2880
Adress requested changes and use final where possible in SoundcloudStreamExtractor
...
This commit moved the HLS parsing task to a separate method, did little performance improvements and used final where possible in the SoundcloudStreamExtractor file.
2021-03-14 17:54:17 +01:00
cbacd3c0a5
Add a check to don't show MP3 128kbps stream twice and catch IOException when fetching the HLS Manifest
...
If a progressive stream is present in the transcodings, it's unnecessary to show twice an MP3 128kbps stream so if this is the case, the MP3 HLS stream will be not added to the audioStreams, else it will.
This commit also catch fetching errors in HLS manifests parsing and don't add the corresponding stream if an error occurs.
2021-03-14 17:54:16 +01:00
26f1b4e7dc
Support SoundCloud HLS by using a workaround
...
This commit tries to support SoundCloud HLS streams by parsing M3U manifests, get the last segment URL (in order to get track length) and request a segment URL equals to track's duration so it's a single URL.
2021-03-14 17:54:14 +01:00
2e57a8f24f
[Bandcamp] Fix link handler acceptance behaviour
...
* Test for bandcamp footer instead of meta tag (which is not present on
all pages)
* Accept links to /music, not just /releases
* Correctly handle uppercase URLs
2021-03-14 09:51:30 +01:00
70d9e389b9
[Bandcamp] Add tests for external bandcamp artists
...
Necessary, because the external pages tested before were converted to bandcamp.com pages. See f9d06252f2
2021-03-13 20:51:59 +01:00
f9d06252f2
[Bandcamp] Fix tests
2021-03-13 17:42:39 +01:00
7e6f464407
Merge pull request #509 from TiA4f8R/soundcloud-improvements
...
Add new exceptions to be able to display different error messages in apps
2021-03-07 13:51:15 +01:00
ac51134aed
Merge pull request #546 from XiangRongLin/unignore2
...
Unignore more tests
2021-03-07 13:15:24 +01:00
22fa131922
Merge branch 'dev' into bandcamp
2021-03-05 19:45:37 +01:00
e55284bb8f
Code style improvements
...
Use final where possible in YoutubeStreamExtractor and do some other code style improvements
2021-03-05 16:38:53 +01:00
d0a1041afd
Prevent NullPointerException and use equalsIgnorecase
2021-03-05 16:38:52 +01:00
890cbba625
Update PaidContent exception
2021-03-05 16:38:51 +01:00
da3cfa967d
Handle age-restricted videos
2021-03-05 16:38:50 +01:00
448b68700c
Use final in exceptions and add AgeRestrictedException
...
AgeRestrictedException will be thrown only if the reason message equals to "Sign in to confirm your age" and if the age limit is 18.
2021-03-05 16:38:49 +01:00
22e6f33f0a
Ignore SoundCloudGeoRestrictedTest until HLS stream extraction is added
...
The ContentNotSupportedException is thrown because no supported audio streams where extracted. However, SoundCLoud does not check, whether there are any streams available.
This commit should be reverted in #526
2021-03-05 16:38:47 +01:00
21158744b1
Add expectedDescriptionIsEmpty() to DefaultStreamExtractorTest
...
Also check if related streams are empty if they are expected to be empty.
2021-03-05 16:38:46 +01:00
f15d7837a1
Fix SoundCloud tests
2021-03-05 16:38:45 +01:00
4e45aef2b3
Fix SoundCloud CreativeCommonsPlaysWellWithOthers test
2021-03-05 16:38:43 +01:00
59d6d3f04e
Fix YouTube tests
2021-03-05 16:38:42 +01:00
3925204658
Add tests and rename SoundcloudGoPlusException to SoundcloudGoPlusContentException
2021-03-05 16:38:40 +01:00
771bb1a2cb
Add three new exceptions for unavailable YouTube contents
...
These exceptions are thrown on a test with the error messages text, because YouTube returns only "UNPLAYABLE" status in most error cases.
Tests are based with English strings, so changing the lang used by
extractor will throw the generic exception (ContentNotAvailableException).
2021-03-05 16:38:39 +01:00
35325d980d
Add GeographicRestrictionException and SoundCloudGoPlusException in NewPipe Extractor to be able to display different error messages
...
This commit adds two new exceptions in NewPipe Extractor: GeographicRestrictionException and SoundCloudGoPlusException (which extend to ContentNotAvailableException). These exceptions allow showing different error messages to user when a content isn't available in his/her/its country (only used for now by SoundCloudStreamExtractor) or when the content is a SoundCloud Go+ track.
2021-03-05 16:38:37 +01:00
bc0cda68d1
Merge pull request #570 from TiA4f8R/fix-yt-channels-playlists
...
Fix YouTube channels contination
2021-03-05 15:26:49 +01:00
a3c6fceef5
Merge pull request #573 from B0pol/comments-performance
...
[youtube] improve comments extraction performance
2021-03-05 14:46:27 +01:00
ff5273b882
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeCommentsExtractor.java
...
Co-authored-by: Tobi <TobiGr@users.noreply.github.com>
2021-03-05 14:39:01 +01:00
ec1127dd6a
Merge pull request #571 from TeamNewPipe/soundcloud_client_id
...
[SoundCloud] Update client_id and fix extraction
2021-03-05 14:21:14 +01:00
ed850d0688
[youtube] improve comments extraction performance
...
- do not parse responseBody twice for continuation
instead try to get commentsTokenInside with the new pattern ("sectionListRenderer")
and try again with the old pattern ("commentSectionRenderer") on failure
- do not unescape responseBody multiple times
-> parse responseBody less times
2021-03-05 13:37:08 +01:00
440a808b8a
Update mocks for channels
2021-03-05 13:13:36 +01:00
05213175cd
minor edit, made variable final
2021-03-05 02:49:15 -05:00
8bceb57a6c
formatting fix
2021-03-05 02:46:06 -05:00
ae48bdea4c
Updated to reflect variable name change
2021-03-05 02:40:36 -05:00
b27efdc9a4
Remove clickTrackingParams in getNextPage method of YoutubeChannelExtractor
...
Because it's unused with the new request.
Co-authored-by: bopol <bopol@e.email>
2021-03-04 22:50:20 +01:00
2b671b15ce
[YouTube] Use new continuation API for channels
...
Co-authored-by: TiA4f8R <74829229+tia4f8r@users.noreply.github.com>
2021-03-04 21:47:16 +01:00
33594d4aff
[SoundCloud] Fix client id extraction
...
SoundCloud changed their Javascript structure. That caused the client_id to be in the middle of the file (at byte ~43000 ). To have a little buffer, we now fetch the first 50000 bytes.
2021-03-04 21:32:34 +01:00
fb2f37d223
[SoundCloud] Update hardcoded client id
2021-03-04 21:25:41 +01:00
506cc5fbe9
regenerate mocks
2021-03-04 19:01:58 +01:00
03b00ff1d6
Move toJsonArray and toJsonObject to JsonUtils
2021-03-04 18:58:51 +01:00
668b080b3a
Use dynamic key.
2021-03-04 08:45:14 +01:00
43b46bd408
Use new youtube continuations api for playlists
...
Requires sending a POST request instead of GET.
clientName and clientVersion, which were required as headers previously now need to be part of the request payload.
continuation id also needs to be part of request body.
quick and dirty solution.
2021-03-03 19:49:26 +01:00
beb05bd05c
Replace test url, because playlist was deleted
2021-03-03 19:46:14 +01:00
e13e237392
Fix typo and reword some explanations
2021-02-26 17:48:03 +01:00
a1688fe953
Move BandcampExtractorHelper.getJsonData(String, String) to JsonUtils
2021-02-21 13:51:12 +01:00
70814dcfef
Fix Utils.nonEmptyAndNullJoin
...
When using the index here, it the index needs to be decremented once an element is removed. To cirecumvent this, the native Collections.removeIf() method is used.
2021-02-21 12:51:47 +01:00
b9e8ee8450
Rename BandcampExtractorHelper.smartConcatenate(String[], String) to Utils.nonEmptyAndNullJoin(String, String[])
2021-02-21 12:35:14 +01:00
c07db80ef0
Add BASE_URL and BASE_API_URL to BandcampExtractorHelper
2021-02-21 12:15:45 +01:00
adde4332d1
Code improvements
2021-02-20 23:58:54 +01:00
42bdd8adc5
Update to fix failing mock tests.
2021-02-20 17:14:41 +05:30
e8bccfaf5d
Change UA to privacy.resistFingerprinting.
2021-02-20 17:14:41 +05:30
0c831afa7b
use DownloaderFactory instead of DownloaderTestImpl for search ChannelVerified
2021-02-20 11:41:30 +01:00
ea120a4637
remove print stacktrace
...
use interface
2021-02-20 00:23:45 +01:00
adf9d7d10f
Add reason field to MockOnly
...
This enforces developers to document why a test is skipped
2021-02-19 18:54:18 +01:00
ea52030613
Add MockOnlyRule to allow skipping specific tests based on downloader
2021-02-19 18:54:17 +01:00
02920fafa8
Add isUploaderVerified()
2021-02-19 15:51:23 +01:00
91e9309486
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-02-19 13:37:45 +01:00
fa61b864f2
Code improvements
2021-02-19 13:37:24 +01:00
98268e351c
Move radio URL check into a function
2021-02-19 13:35:44 +01:00
5333d8a98b
Add more data.
2021-02-19 00:55:03 +05:30
1a322ad8ed
Add uploader verified by service extraction
2021-02-18 16:36:40 +01:00
c24afa2cbb
Permalink URL.
2021-02-18 20:16:53 +05:30
8d54401233
Add fps field as well.
2021-02-18 20:16:53 +05:30
5b70645d85
Add data to muxed streams aswell.
2021-02-18 20:16:53 +05:30
e975d33fbe
Change jdoc / comments.
2021-02-18 20:16:53 +05:30
f31b2a68fd
use getters/setters.
2021-02-18 20:16:53 +05:30
d4945ac55c
Constructor cleanup.
2021-02-18 20:16:52 +05:30
3f6a601be8
Fix jdoc / comments.
2021-02-18 20:16:52 +05:30
a7c9905183
Fix requested changes.
2021-02-18 20:16:52 +05:30
0c0f2d74bc
Make fields private
...
Co-authored-by: bopol <bopol@e.email>
2021-02-18 20:16:52 +05:30
d1054338d3
Fix error with livestreams.
2021-02-18 20:16:52 +05:30
11eb4932f4
Add data to respective classes.
2021-02-18 20:16:52 +05:30
525e345ed8
remove duplicate field.
2021-02-18 20:16:52 +05:30
e844d2aed3
Add dash fields to the ItagItem.
2021-02-18 20:16:52 +05:30
64f6b0478f
Merge pull request #545 from XiangRongLin/comment_reset
...
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-18 15:14:03 +01:00
a64579da2f
Unignore some tests that work on Linux
2021-02-17 20:31:26 +01:00
b99cb8624f
Unignore PublicBroadcasterTest
...
Tests only work on linux, mocks have to be generated on linux too
2021-02-17 20:31:26 +01:00
37366c65ca
Remove encoding of '&' to '&' in test and unignore it
2021-02-17 20:31:26 +01:00
c327922cc0
Add explanations to broken tests
2021-02-17 20:31:26 +01:00
26d50aca05
Add comment explaining YoutubeParsingHelper.resetClientVersionAndKey
2021-02-17 19:21:39 +01:00
54aa5b3042
Use propper structure in KioskExtractors
...
Made BandCampRadioExtractor a Kiosk which holds StreamInfoItems and not InfoItems.
2021-02-16 21:09:42 +01:00
54b8e54f80
Fix potential NPE
2021-02-16 20:59:13 +01:00
16973126a0
Use Collections.emptyList() instead of Collections.EMPTY_LIST or new ArrayList()
2021-02-16 20:25:48 +01:00
84380e4c3a
Merge pull request #536 from B0pol/watchendpoints
...
support /watch/, /v/ and /w/ subpaths
2021-02-16 20:23:27 +01:00
41f689b099
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/linkHandler/YoutubeStreamLinkHandlerFactory.java
...
Co-authored-by: Stypox <stypox@pm.me>
2021-02-16 19:17:10 +01:00
a9594f531f
Merge pull request #543 from XiangRongLin/unignore1
...
Unignore and fix some tests
2021-02-16 18:00:16 +01:00
a28e0226c5
Merge pull request #519 from XiangRongLin/better_logging_error_handling
...
Better logging, error handling, doc for mock tests
2021-02-16 15:06:06 +01:00
05cb22c654
Unignore Gronkh.testDescription
...
Remove special character ü from assertion
2021-02-16 08:59:35 +01:00
d9b4c2779f
Unignore YoutubeSearchQHTest.testRegularValues
2021-02-16 08:59:35 +01:00
137272b7a4
Unignore YoutubeCommentsExtractorTest.EmptyComment
...
Mock files were missing
2021-02-16 08:59:34 +01:00
5bf9fddba9
Code improvements
2021-02-15 22:45:17 +01:00
e062c8cb0d
Merge branch 'dev' into bandcamp
2021-02-15 22:09:41 +01:00
7a3d9bdb7d
Description: add .equals() and .hashCode()
2021-02-15 18:51:47 +01:00
557934cb17
use EMPTY_STRING
2021-02-15 18:51:46 +01:00
9ca52ca68a
use only one constant UTF-8
2021-02-15 18:47:54 +01:00
50e5718bd1
Merge pull request #518 from XiangRongLin/generate_mocks
...
Use and generate mocks for youtube tests
2021-02-15 06:39:41 +01:00
d7486f251a
Merge pull request #538 from B0pol/soundcloud-georestriction
...
Soundcloud: fix charts extraction when georestricted
2021-02-14 23:52:27 +01:00
432c68cdec
Update okhttp from 3.12.11 to 3.12.13
...
Changelog https://square.github.io/okhttp/changelog_3x/#version-31213
2021-02-14 23:44:19 +01:00
a88b2e6af4
Undo formatting of YoutubeStreamExtractorDefaultTest
2021-02-14 14:57:23 +01:00
8d89c82caa
Soundcloud: fix charts extraction when georestricted
2021-02-13 22:28:31 +01:00
ee2012c948
rename folder to subpath
2021-02-13 22:25:46 +01:00
8db40db4de
use substring instead of split
2021-02-13 22:20:05 +01:00
81317cf6bc
youtube: support /w/ folder
2021-02-13 22:20:05 +01:00
f15c0fcfed
support /watch/ and /v/ folders
2021-02-13 22:20:05 +01:00
971128c464
Ignore failing tests
2021-02-13 20:28:03 +01:00
92dea0806a
Regenerate json files for youtube tests
2021-02-13 20:28:03 +01:00
d6b9930c18
Use DownloaderFactory in YoutubeCommentsExtractorTest.Pinned
2021-02-13 20:12:39 +01:00
b555f38fa5
Use real downloader for YoutubeSubscriptionExtractorTest
...
It doesn't make network requests anyway
2021-02-13 20:12:39 +01:00
f352b0c7af
YoutubeMusicSearchExtractorTest can't be used with mocks
...
Requires further research
2021-02-13 20:00:51 +01:00
f45b5610bf
Unignore YoutubeChannelLocalizationTest
...
With mocks time is not a concern anymore
2021-02-13 20:00:51 +01:00
19737e06ee
Use DownloaderFactory in YoutubeChannelExtractorTest
...
KurzgesagtAdditional needed to be extracted because it ran a duplicated request.
2021-02-13 20:00:51 +01:00
84eb4b30c9
Use DownloaderFactory in YouTube tests
2021-02-13 20:00:51 +01:00
b43f3474f5
Make resetClientVersionAndKey public
2021-02-13 20:00:49 +01:00
2859c7ba5c
add comment explaining why .substring(23)
2021-02-13 12:10:41 +01:00
73353996d7
fix more tests
2021-02-12 23:18:38 +01:00
a931e31239
fix tests
2021-02-12 22:36:18 +01:00
d116680ea3
Merge branch 'master' into dev
2021-02-12 22:26:35 +01:00
b0f356dd76
YT channels: fix continuation
2021-02-12 14:39:09 +01:00
fcdb50b825
make getInfo from Extractor public
...
In StreamInfo and CommentsInfo
2021-02-09 14:58:43 +01:00
44c54d403a
fix Invidious channels being accepted by YoutubeStreamLinkHandlerFactory ( #527 )
...
fixes https://github.com/TeamNewPipe/NewPipeExtractor/issues/524 , see it for more details
2021-02-05 18:57:58 +01:00
1414a6f178
Fix extraction of YouTube chapters ( #531 )
...
YouTube changed the name of the chapters in the JSON from engagement-panel-macro-markers to engagement-panel-macro-markers-description-chapters, so extracting chapters doesn't work.
2021-02-03 22:07:34 +01:00
bfa639950e
Merge pull request #528 from Stypox/ytmusic-0
...
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 23:23:23 +01:00
d728c4fad0
Remove unused imports
...
I searched for "StandardCharsets" to prevent possible bugs with Android KitKat. The import is unused and is therefor removed.
There are other usages of the StandardCharsets class in the test implementation, but those are not relevant.
2021-01-29 23:16:40 +01:00
a64dfd7343
[YouTube Music] Fix extracting search item view/subscriber count when = 0
2021-01-29 16:44:24 +01:00
2662ceca22
rename getHeartedByCreator and getPinned to isPinned and isHeartedByCreator
2021-01-23 13:06:07 +01:00
66e4eb2f96
extract whether the comment is pinned
2021-01-22 23:59:22 +01:00
67ddfefdc6
keep invidious instances list in the same order
...
as suggested by @TobiGr
2021-01-22 20:58:14 +01:00
53d3f7989b
update invidious instances list
2021-01-22 19:30:25 +01:00
88e4c8667b
Add comments to RecordingDownloader about usage
2021-01-17 19:53:20 +01:00
eecfe09f2c
Check for non-existent folder in MockDownloader
2021-01-17 19:52:58 +01:00
07570554b6
Fix build and optimize imports
2021-01-15 21:55:40 +01:00
78c2113094
Merge remote-tracking branch 'origin/dev' into bandcamp
2021-01-15 21:49:58 +01:00
b9ba95614b
Merge pull request #514 from XiangRongLin/test_jvm_property
...
Pass on gradle JVM system property to test JVM
2021-01-15 20:44:23 +01:00
a6cfe11ae9
Pass on gradle JVM system property to test JVM
2021-01-15 20:37:14 +01:00
6888e7773f
Ignore all failing tests
2021-01-15 20:11:40 +01:00
7c774c84ce
Merge pull request #505 from TeamNewPipe/meida.ccc.de_search_fix
...
[media.ccc.de] Fix NPE in search results if they contain a future talk
2021-01-15 14:16:59 +01:00
c2c4d97efd
Merge pull request #482 from XiangRongLin/mock_mix_pl_test
...
Mock mix pl tests
2021-01-15 12:54:48 +01:00
4f81d9d689
Regenerate mock files
2021-01-15 09:51:23 +01:00
951159f75f
Merge pull request #504 from TeamNewPipe/media.ccc.de_recent_order
...
[media.ccc.de] Recent kiosk: order streams by upload date
2021-01-14 23:31:09 +01:00
cc51c5f6a3
Merge pull request #507 from vkay94/add-interval-to-stream-frames
...
Add durationPerFrame to Frameset
2021-01-14 20:27:40 +01:00
6ca7123d5d
Add comment for usage of YoutubeParsingHelper.resetClientVersionAndKey
2021-01-14 20:01:52 +01:00
11dcfe638b
Add durationPerFrame to Frameset and getFrameBoundsAt method
2021-01-14 20:01:06 +01:00
c2ff6723d7
Merge pull request #511 from B0pol/errors
...
Fix PeerTube tests by changing instance
2021-01-14 19:15:22 +01:00
6773ec8699
Merge pull request #510 from B0pol/heartedbycreator
...
comments: add heartedByUploader and extract it for YouTube
2021-01-12 19:00:04 +01:00
0c4836c800
[media.ccc.de] Recent kiosk: Add comment clarifying "length" and "duration" fields have the same value
2021-01-12 18:48:42 +01:00
4af50c6870
Use @link for downloader
2021-01-12 09:41:57 +01:00
3455f0f23c
Add documentation to testing with mocks
2021-01-12 09:37:48 +01:00
5292cc1636
[peertube] test: remove usage of peertube.mastodon.host
...
It's down for months
2021-01-11 19:46:27 +01:00
f3c22da618
comments: add heartedByCreator and extract it for youtube
2021-01-11 18:48:08 +01:00
63c237db41
Only create directories if they do not exist
2021-01-11 07:57:11 +01:00
e82cb1efea
Remove dependency to commons-io
2021-01-10 20:50:38 +01:00
1ea6c6ce54
Prefix mock file from RecordingDownloader with "generated_mock_"
...
Only read those files in MockDownloader
2021-01-10 20:46:42 +01:00
f91916c017
Remove unnecessary file object creation
2021-01-10 20:39:22 +01:00
255c726f20
Add resource path base to DownloaderFactory
2021-01-10 20:27:34 +01:00
35e299759e
Add method to allow resetting youtube client version and key
...
This is needed so that a request is made for each test class when running multiple at once. This way RecordingDownloader records all necessary requests.
This works as long as tests are run sequentially and not in parallel.
2021-01-10 20:24:50 +01:00
f447a7a450
fix import due to downloader package move
2021-01-10 19:49:14 +01:00
f8aa989c42
Add generated json and manually copy client_version.json
2021-01-10 19:38:47 +01:00
285c26eafa
Adjust YoutubeMixPlaylistExtractorTest to use DownloaderFactory
2021-01-10 19:38:41 +01:00
e6e8e39def
Add DownloaderFactory to return a specific downloader based on 2 variables.
...
If the system property 'downloader' is set that use that specific downloader. This is used from gradle by appending `-Ddownloader=ABCD to the command.
ABCD is one of DownloaderType.
The other variable is the static property `DEFAULT_DOWNLOADER` in DownloaderFactory, which can be easily changed as needed inside the IDE according to development needs`.
Normal workflow would be to first use the recording downloader and afterwards only use mocks, if the requests are always staying the same.
2021-01-10 19:36:24 +01:00
7c40fb8bf7
Add additional downloader implementations
...
RecordingDownloader relies on the real downloader and saves the request/response pair into a json file.
MockDownloader uses json files from above and mocks responses for specific requests.
2021-01-10 19:36:24 +01:00
1bcb9c76a7
Generate equals and hashCode for Request
2021-01-10 19:36:24 +01:00
4dad3d60d6
Move DownloaderTestImpl into downloader package
2021-01-10 19:36:21 +01:00
fe3902c669
Un-comment out 360p webm
2021-01-08 19:43:03 +01:00
2d93b23723
[media.ccc.de] Show conference title in search results instead of slug
2021-01-08 14:15:25 +01:00
52f3bd15f2
[media.ccc.de] Add conference title to "recent" kiosk
...
The "conference_title" field was added upstream recently (see https://github.com/voc/voctoweb/issues/495 )
2021-01-08 12:04:39 +01:00
ab62464e86
Merge pull request #495 from B0pol/sepiasearch
...
[PeerTube] Implement sepia search
2021-01-06 11:06:36 +01:00
0bcea9c2cd
add JDoc
2021-01-01 14:37:32 +01:00
0e45c25ea9
Update extractor/src/main/java/org/schabi/newpipe/extractor/services/peertube/extractors/PeertubeStreamInfoItemExtractor.java
...
Co-authored-by: Tobias Groza <TobiGr@users.noreply.github.com>
2021-01-01 14:20:47 +01:00
fae67fbd45
Merge remote-tracking branch 'origin/master' into dev
2021-01-01 13:06:11 +01:00
d63d20f943
Fix JDoc
2020-12-31 18:05:40 +01:00
2cbc3ccb1a
[media.ccc.de] Fix live stream test if there are no current live streams
2020-12-31 17:03:23 +01:00
a896ec41b9
[media.ccc.de] Use Pattern for getting the ive stream id
2020-12-31 17:03:23 +01:00
5dc9a76e3c
[media.ccc.de] Recent kiosk: order streams by upload date
2020-12-30 18:38:09 +01:00
edf8dd0e92
[media.ccc.de] Fix NPE in search results if they contain a future talk
2020-12-30 18:34:28 +01:00
3c8c8e7307
Merge pull request #497 from TiA4f8R/fix-player-js-url-redirect
...
Fix YouTube player JavaScript URL redirection
2020-12-27 18:04:40 +01:00
d30caeb091
Fix YouTube player JavaScript URL redirection
2020-12-27 15:55:03 +01:00
deb9af7bf5
[media.ccc.de] live kiosk - display info of the current talk
2020-12-27 15:52:09 +01:00
b4e1913971
[media.ccc.de] Play live streams
2020-12-27 13:25:48 +01:00
80f4d42226
[Draft] Add live extractor
2020-12-27 11:01:22 +01:00
ed9402c002
[media.ccc.de] Escape "." in id pattern
2020-12-27 11:01:10 +01:00
674f2227f3
make url pattern static
2020-12-27 01:27:37 +01:00
090dade5b3
[media.ccc.de] Add short test for recent kiosk
2020-12-27 01:03:08 +01:00
0cfefe222a
Revert "[media.ccc.de] Get conference name from API URL"
...
This reverts commit d1a0686d2f
2020-12-26 16:55:44 +01:00
d1a0686d2f
[media.ccc.de] Get conference name from API URL
2020-12-26 13:15:03 +01:00
41699798fe
[media.ccc.de] Use web URL for uploaderUrl instead of API URL in recent
2020-12-26 12:47:32 +01:00
b2154c6e87
[media.ccc.de] Fix conferences kiosk test
2020-12-25 23:55:45 +01:00
ed6ae329b3
[media.ccc.de] Add "recent" kiosk
2020-12-25 22:26:58 +01:00
b5e50cc9fb
Merge pull request #484 from TeamNewPipe/yt_,music_search
...
Fix YouTube Music search
2020-12-25 18:30:25 +01:00
d9e2da53c3
Replace explicit string checks whether a playlist íd is a certain YouTube Mix type with calling the dedicated methods
2020-12-25 15:00:34 +01:00
abee0a8df1
Fix typos in YoutubePlaylistLinkHandlerFactory
2020-12-24 10:34:21 +01:00
3033c0b993
[YouTube] Music Mix: Fix playlist not being accepted
...
Regression introduced by YouTube Mix support (#280 )
2020-12-23 21:08:36 +01:00
b62144b49d
[YouTube] Music: Fix search extraction caused by changed JSON response
2020-12-23 21:08:36 +01:00
c9d9bd1e24
Fix typo in ListLinkHandlerFactory#getUrl
2020-12-23 21:05:58 +01:00
3ae924a7f1
Implement sepia search
2020-12-23 15:07:55 +01:00
942d840624
Fix typo in AffiliateService: rename "NO_AFILIATE" to "NO_AFFILIATE"
2020-12-23 12:50:45 +01:00
865c42e273
Fix line break
2020-12-23 12:49:50 +01:00
c9e9953bb0
[Bandcamp] Fix channel link handler factory
2020-12-22 20:02:53 +01:00
41a8ed625d
Make some vars final and add annotations to methods
2020-12-22 18:19:26 +01:00
bc6de14952
Extract stream and search meta info for YouTube
...
Add method to extract Google webcache URLs.
2020-12-22 18:19:26 +01:00
50903730b1
[Bandcamp] Fix accepting HTTP URLs
2020-12-22 08:53:10 +01:00
litetex
Stypox
XiangRongLin
TiA4f8R
FireMasterK
TobiGr
dependabot[bot]
Abdu Ameen
Mohammed Anas
bopol
opusforlife2
talanc
magicfelix
gardenapple
TacoTheDank
Hassan Igbaria
Fynn Godau
golfinq
vkay94