Commit Graph

2373 Commits

Author SHA1 Message Date
Stypox c336bd58a5
Merge pull request #879 from TeamNewPipe/dependabot/gradle/org.junit-junit-bom-5.9.0
Bump junit-bom from 5.8.2 to 5.9.0
2022-08-03 21:06:20 +02:00
dependabot[bot] 325af31e5f
Bump gson from 2.9.0 to 2.9.1
Bumps [gson](https://github.com/google/gson) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.9.0...gson-parent-2.9.1)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-08-01 09:26:14 +00:00
dependabot[bot] d905636021
Bump junit-bom from 5.8.2 to 5.9.0
Bumps [junit-bom](https://github.com/junit-team/junit5) from 5.8.2 to 5.9.0.
- [Release notes](https://github.com/junit-team/junit5/releases)
- [Commits](https://github.com/junit-team/junit5/compare/r5.8.2...r5.9.0)

---
updated-dependencies:
- dependency-name: org.junit:junit-bom
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-27 09:20:12 +00:00
Isira Seneviratne 1af6b8eedb Use Collections.singletonList(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne ff60e05c76 Use Collections.singletonMap(). 2022-07-27 07:35:57 +05:30
Isira Seneviratne 682a4263e5 Use Objects.requireNonNull(). 2022-07-27 06:55:26 +05:30
Stypox 8c5f014a6f
Merge pull request #869 from mhmdanas/add-workflow-permissions
Use minimum required permissions for GitHub workflows
2022-07-13 18:58:32 +02:00
Stypox 954a294e27
Merge pull request #870 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.2
Bump jsoup from 1.15.1 to 1.15.2
2022-07-13 17:48:06 +02:00
Mohammed Anas f57049d2c0
Use temurin instead of adopt (#868)
See
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/.
2022-07-10 18:38:08 +03:00
Stypox 5bd7bf20cc
Merge pull request #867 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.1
Bump spotbugs-annotations from 4.7.0 to 4.7.1
2022-07-06 14:23:44 +02:00
Stypox 5ab74b3631
Merge pull request #857 from FireMasterK/video-title
Get original untranslated title for YouTube
2022-07-06 10:26:45 +02:00
dependabot[bot] 122365005a
Bump jsoup from 1.15.1 to 1.15.2
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.15.1 to 1.15.2.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.15.1...jsoup-1.15.2)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-07-04 09:38:16 +00:00
mhmdanas 414186cff5 Use minimum required permissions for GitHub workflows
This reduces the attack surface if the workflows are ever compromised.
2022-07-03 23:55:28 +03:00
dependabot[bot] e9b4be3e3c
Bump spotbugs-annotations from 4.7.0 to 4.7.1
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.7.0 to 4.7.1.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.7.0...4.7.1)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-28 09:15:07 +00:00
Stypox 5219a705ba
Merge pull request #864 from AudricV/fetch-android-client-for-ended/post-livestreams
[YouTube] Fetch the ANDROID client for ended/post livestreams
2022-06-24 14:55:20 +02:00
AudricV 090debd83b
[YouTube] Fetch the ANDROID client for ended/post livestreams
The ANDROID client was only fetched for video contents, where it can be useful on ended/post livestreams, if the n parameter of the WEB client cannot be decrypted, to avoid throttling issues (because the WEB client was only used before for ended/post livestreams).

It also provides an exclusive 48kbps M4A audio format in the adaptiveFormats array of the JSON player response, like other mobile clients (which can be also extracted from the response of the DASH manifest URL returned into the WEB client player's response, but the DASH manifest is not used by the extractor).

A note about non-fatality of fetching or parsing issues of the ANDROID and IOS clients has been added.
2022-06-21 18:53:49 +02:00
litetex a26bcc55c4
Merge pull request #845 from TeamNewPipe/dependabot/gradle/com.github.spotbugs-spotbugs-annotations-4.7.0
Bump spotbugs-annotations from 4.6.0 to 4.7.0
2022-06-19 15:36:55 +02:00
dependabot[bot] 424eb1c559
Bump spotbugs-annotations from 4.6.0 to 4.7.0
Bumps [spotbugs-annotations](https://github.com/spotbugs/spotbugs) from 4.6.0 to 4.7.0.
- [Release notes](https://github.com/spotbugs/spotbugs/releases)
- [Changelog](https://github.com/spotbugs/spotbugs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/spotbugs/spotbugs/compare/4.6.0...4.7.0)

---
updated-dependencies:
- dependency-name: com.github.spotbugs:spotbugs-annotations
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:29:18 +00:00
litetex 2712c3d788
Merge pull request #847 from TeamNewPipe/dependabot/gradle/org.jsoup-jsoup-1.15.1
Bump jsoup from 1.14.3 to 1.15.1
2022-06-19 15:28:43 +02:00
dependabot[bot] 281d2b9f81
Bump jsoup from 1.14.3 to 1.15.1
Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.1.
- [Release notes](https://github.com/jhy/jsoup/releases)
- [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES)
- [Commits](https://github.com/jhy/jsoup/compare/jsoup-1.14.3...jsoup-1.15.1)

---
updated-dependencies:
- dependency-name: org.jsoup:jsoup
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-06-19 13:15:40 +00:00
litetex f775155d25
Merge pull request #846 from litetex/remove-unused-methods
Remove unused methods
2022-06-19 15:12:15 +02:00
litetex 9d625dd75f
Merge pull request #860 from gliptak/patch-1
Correct PeertubePlaylistExtractorTest unit tests
2022-06-19 15:09:27 +02:00
Gábor Lipták 28c9340d69
Correct unit tests 2022-06-18 11:18:38 -04:00
Stypox 1b51eab664
Merge pull request #859 from AudricV/delivery-methods-fixes-and-improvements
Fix extraction of some properties in ItagItems for YouTube livestreams and post-live streams and remove completely SoundCloud HLS workaround
2022-06-17 15:46:13 +02:00
AudricV 301a795ed3
[SoundCloud] Remove completely workaround for HLS streams
SoundCloud is currently removing this workaround completely, so there is no need to keep it, because it impacts the loading time (a HLS playlist was downloaded and parsed).
2022-06-16 12:12:54 +02:00
AudricV e960a417ec
[YouTube] Fix extraction of fps, audioSampleRate and audioChannels fields for ItagItems of live streams and post live streams
These values were only set before for video streams.

A fallback for the audio channels count has been added, in order to prevent exceptions when generating DASH manifests of audio streams: the fallback value is 2, because most audio streams on YouTube have 2 audio channels.
2022-06-16 12:12:54 +02:00
Stypox c8a77da2ab
Merge pull request #810 from TiA4f8R/delivery-methods-v2
Support delivery methods other than progressive HTTP
2022-06-02 22:44:24 +02:00
Kavin 7635aeed2c
Get original untranslated title for YouTube. 2022-06-02 09:57:52 +01:00
TiA4f8R 287d1dfd63
[SoundCloud] Use the HLS delivery method for all streams and extract only a single stream URL from HLS manifest for MP3 streams
SoundCloud broke the workaround used to get a single file from HLS manifests for Opus manifests, but it still works for MP3 ones.

The code has been adapted to prevent an unneeded request (the one to the Opus HLS manifest) and the HLS delivery method is now used for SoundCloud MP3 and Opus streams, plus the progressive one (for tracks which have a progressive stream (MP3) and for the ones which doesn't have one, it is still used by trying to get a progressive stream, using the workaround).

Streams extraction has been also moved to Java 8 Stream's API and the relevant test has been also updated.
2022-05-29 19:08:18 +02:00
Stypox b3c620f0d8
Apply code review and Streams rework 2022-05-28 12:00:58 +02:00
Stypox d652e05874
[MediaCCC] Fix comments about containsSimilarStream 2022-05-28 12:00:58 +02:00
Stypox 044639c32b
Solve some review comments 2022-05-28 12:00:57 +02:00
litetex c33d392958
Fixed typo XEE → XXE (Xml eXternal Entity attack)
See also:
https://en.wikipedia.org/wiki/XML_external_entity_attack
https://owasp.org/www-community/vulnerabilities/XML_External_Entity_(XXE)_Processing
2022-05-28 12:00:57 +02:00
TiA4f8R fffbbee7f3
[YouTube] Add missing Nonnull annotations in getCache method of YouTube DASH manifest creators 2022-05-28 12:00:56 +02:00
TiA4f8R f7b1515290
[YouTube] Refactor DASH manifests creation
Move DASH manifests creation into a new subpackage of the YouTube package, dashmanifestcreators.
This subpackage contains:

- CreationException, exception extending Java's RuntimeException, thrown by manifest creators when something goes wrong;
- YoutubeDashManifestCreatorsUtils, class which contains all common methods and constants of all or a part of the manifest creators;
- a manifest creator has been added per delivery type of YouTube streams:
  - YoutubeProgressiveDashManifestCreator, for progressive streams;
  - YoutubeOtfDashManifestCreator, for OTF streams;
  - YoutubePostLiveStreamDvrDashManifestCreator, for post-live DVR streams (which use the live delivery method).

Every DASH manifest creator has a getCache() static method, which returns the ManifestCreatorCache instance used to cache results.

DeliveryType has been also extracted from the YouTube DASH manifest creators part of the extractor and moved to the YouTube package.

YoutubeDashManifestCreatorTest has been updated and renamed to YoutubeDashManifestCreatorsTest, and YoutubeDashManifestCreator has been removed.

Finally, several documentation and exception messages fixes and improvements have been made.
2022-05-28 12:00:56 +02:00
TiA4f8R f17f7b9842
Apply requested changes in YoutubeParsingHelper 2022-05-28 12:00:55 +02:00
TiA4f8R 301b9fa024
Remove hashCode and equals methods overrides of Stream classes 2022-05-28 12:00:55 +02:00
TiA4f8R 2f3920c648
[YouTube] Return approxDurationMs value from YouTube's player response in ItagItems generated
This change allows to build DASH manifests using YoutubeDashManifestCreator with the real duration of streams and prevent potential cuts of the end of progressive streams, because the duration in YouTube's player response is in seconds and not milliseconds.
2022-05-28 12:00:54 +02:00
TiA4f8R 4158fc46a0
[Bandcamp] Fix regression of Opus radio streams extraction
When moving opus-lo into a constant, opus-lo was renamed to opus_lo and was only used if no MP3 stream was available (which was not the case before the changes in BandcampRadioStreamExtractor related to the addition of the support of all delivery methods), so these changes removed the ability to get Opus streams of Bandcamp radios.

This commit reverts this unwanted change.
2022-05-28 12:00:54 +02:00
TiA4f8R 54d323c2ae
Fix Checkstyle issue in YoutubeDashManifestCreator 2022-05-28 12:00:53 +02:00
Stypox 2321822844
Rename Stream's baseUrl to manifestUrl 2022-05-28 12:00:53 +02:00
Stypox cfc13f4a6f
[YouTube] Reduce exception generation code and move several attributes of MPD documents into constants
Rename YoutubeDashManifestCreationException to CreationException.
Also use these constants in YoutubeDashManifestCreatorTest.
2022-05-28 12:00:53 +02:00
Stypox 00bbe5eb4d
[YouTube] Make exception messages smaller 2022-05-28 12:00:52 +02:00
Stypox 4da05afe11
[YouTube] Inline collectSegmentsData in YoutubeDashManifestCreator 2022-05-28 12:00:52 +02:00
Stypox 3708ab9ed5
[YouTube] Refactor YoutubeDashManifestCreator
- Remove all of the methods used to access caches and replace them with three caches getters
- Rename caches to shorter and more meaningful names
- Remove redundant @throws tags that just say "if this method fails to do what it should do", which is obvious
2022-05-28 12:00:51 +02:00
Stypox 5c83409039
[YouTube] Rewrite manifest test and rename long methods 2022-05-28 12:00:51 +02:00
Stypox 8226fd044f
[YouTube] Suppress Sonar security warning for XEE 2022-05-28 12:00:50 +02:00
Stypox ba68b8c014
[YouTube] Secure DashManifestCreator against XEE attack
Also remove duplicate lines from javadoc comments, otherwise checkstyle complaints.
XEE prevention is inspired from https://github.com/ChuckerTeam/chucker/pull/201/files
For an explanation of XEE see https://rules.sonarsource.com/java/RSPEC-2755, though the solution reported there causes crashes on Android
2022-05-28 12:00:50 +02:00
Stypox 159d05c91b
Remove unused DashMpdParser (but kept in git history) 2022-05-28 12:00:49 +02:00
Stypox 50272db946
Apply reviews: improve comments, remove FILE, remove Stream#equals(Stream) 2022-05-28 12:00:49 +02:00