From nobody Tue Nov 15 11:21:10 2022 X-Original-To: dev-commits-ports-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NBNzC1BDfz4hgwM; Tue, 15 Nov 2022 11:21:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NBNzC0mWzz4Fp3; Tue, 15 Nov 2022 11:21:11 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1668511271; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4E3yQF5Hk5h2uCapJmFKdve2nbnfDmYD1Wb64OLqH4s=; b=DLbEIQqRDN6840AtOEgBKcC/bBBWoo9eqwwrO39qAUv0PsSRodEJATIH9AJeBDnesqMKHM Wksk/IGR+hu5WBpLfmuSxioD47+9sClM8GjhZIYfvP4odxM/egj/ntFf55d0GdpZ1N76m2 kp8fxXzaYJ1zTAvJYWibf+vNYAvNmUeDw64/usOxChWoto5MCzqQRY41P2ziNfUHQH82Yk S62knylBYPa+FADfNJfyzVTPr1ZxbY2UyyvVsJMoBmXyQu9ePtgyHKS4ZEaqTeF7wanCRz Dl4CLv/5pfCI/oJBTyE8AiUZI+Zo2RaSoS+B5g83eJIhNVco+d6yvXzBvf6N4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1668511271; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=4E3yQF5Hk5h2uCapJmFKdve2nbnfDmYD1Wb64OLqH4s=; b=FuvkRPMca09mVES5UnnFVg/tgv8e3CzLNKxRIfmwhiriMFBZV1fLzpJhlTr4eYYa7qN9el ODpIT2c0opk577cEwzbaevANHPzS7xuxbNGjcORR54jAvp/8k5Opp0m+7s2ra0YDz1Is8R V35AgjAMiGlzj/QFn3aaHBVeNUs2UibJPAjr1zkEmyrYQr74QgKwGMm0LnQQIqFHS9hfB/ T+UEe8QpMZtkLgNyr6RFLb6PLiwSt4dbKczhduPiHh1cNmAHttNtyNv+4HfBKJGZ4M7t3j M6lDDpQx6IeRp0ze47Y2DH7vQ2OXIxmVvSqg19t4pcdsShzmOq8Hkz/qWUsZ+g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1668511271; a=rsa-sha256; cv=none; b=FRGJsswX3oTPHlBgUXszHjtNK8+0ZMCZazWXUS8TBDL/vjXtCChNIxsRJ01sPRQN5tN//V hxm71B4ncdzjfHyxf0pfajdtJ1HxV+GTRjO/LytYyb4jqWhfCe0ULGBBG6DQ2Mi/flf79J qOffQZByp0z4QgK87DIImo+U8VHP4lO7MUEuDB7u/W3EgFP4ylqmuwzjlcJ4/QJfQM0YMr Rnn5nTQRS690qMIQfaItwMxfdVwGbX5/6zCK0gmrxy6qOm7VuOlASjXEiaVO7Djq/WqlWx TiTVEyN2CYKCJZQDi9VpMCkTRFH0vvTI8kft5bfthtqNF/U/EGtf7WDr324OIQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4NBNzB6xpSzL43; Tue, 15 Nov 2022 11:21:10 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 2AFBLACm086298; Tue, 15 Nov 2022 11:21:10 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 2AFBLAGb086297; Tue, 15 Nov 2022 11:21:10 GMT (envelope-from git) Date: Tue, 15 Nov 2022 11:21:10 GMT Message-Id: <202211151121.2AFBLAGb086297@gitrepo.freebsd.org> To: ports-committers@FreeBSD.org, dev-commits-ports-all@FreeBSD.org, dev-commits-ports-main@FreeBSD.org From: Alex Kozlov Subject: git: de0ec19c76ca - main - www/googlebook_dl: Update to 20160213 List-Id: Commit messages for all branches of the ports repository List-Archive: https://lists.freebsd.org/archives/dev-commits-ports-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-ports-all@freebsd.org X-BeenThere: dev-commits-ports-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: ak X-Git-Repository: ports X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: de0ec19c76cac7c563647628a3d1454fd7ccf8d7 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by ak: URL: https://cgit.FreeBSD.org/ports/commit/?id=de0ec19c76cac7c563647628a3d1454fd7ccf8d7 commit de0ec19c76cac7c563647628a3d1454fd7ccf8d7 Author: Alex Kozlov AuthorDate: 2022-11-15 11:15:58 +0000 Commit: Alex Kozlov CommitDate: 2022-11-15 11:20:51 +0000 www/googlebook_dl: Update to 20160213 --- www/googlebook_dl/Makefile | 2 +- www/googlebook_dl/files/googlebook_dl.sh | 45 ++++++++++++-------------------- 2 files changed, 17 insertions(+), 30 deletions(-) diff --git a/www/googlebook_dl/Makefile b/www/googlebook_dl/Makefile index ae9120054e7c..6b82fb26ec65 100644 --- a/www/googlebook_dl/Makefile +++ b/www/googlebook_dl/Makefile @@ -1,5 +1,5 @@ PORTNAME= googlebook_dl -PORTVERSION= 20150711 +PORTVERSION= 20160213 CATEGORIES= www MASTER_SITES= # none DISTFILES= # none diff --git a/www/googlebook_dl/files/googlebook_dl.sh b/www/googlebook_dl/files/googlebook_dl.sh index 7859d5cd7fa5..d00d46116a9f 100644 --- a/www/googlebook_dl/files/googlebook_dl.sh +++ b/www/googlebook_dl/files/googlebook_dl.sh @@ -6,7 +6,7 @@ parse_options() { - local _proxylist + local OPT _proxylist while getopts ap:P:vw: OPT; do # escape meta @@ -61,7 +61,7 @@ usage() echo ' -P pageprefix when numpages specified (*PA, PP, PR, PT)' echo ' -p https://proxy.tld:port,proxy.tld,ip:port | https-proxylist.txt' echo ' -v verbose' - echo ' -w pagewidth (800, *1024, 1280, 1440, 1680, ...)' + echo ' -w pagewidth (800, *1024, 1280, 1440, 1680, ... 2500)' echo exit 1 } @@ -106,22 +106,16 @@ out() get_cookie() { - local cookie_str _return - - # remove old cookie - rm "${cookie}" 2>/dev/null - # get cookie - wget ${optcommon} -U"${ua}" --keep-session-cookies \ - --save-cookies "${cookie}" -O/dev/null \ - "${baseurl}${bookid}&pg=PA1&jscmd=click3" - - # fail if non-zero exitcode returned or cookie has wrong format - _return=$? - cookie_str="$(sed -ne \ - '/^.google.com[[:space:]]/s/^.*\(NID[^=]*=.*\)$/\1/p' "${cookie}")" - if [ ${_return} -ne 0 -o -z "${cookie_str}" ]; then - rm "${cookie}" 2>/dev/null + unset cookie_str + cookie_str=$(wget ${optcommon} -S -U"${ua}" -O/dev/null \ + "${baseurl}${bookid}&pg=PA1&jscmd=click3" 2>&1 | \ + sed -ne '/Set-Cookie:/s/^.*\(NID[^=]*=.*domain=.google.com; HttpOnly\).*$/\1/p') + + # fail only if cookie has wrong format + # don't care about non-zero exitcode, redirection from google.com + # to national googletld can fail, especially if under the proxy + if [ -z "${cookie_str}" ]; then out 'E\n' "cannot get cookie: ${cookie_str}" return 1 fi @@ -146,16 +140,16 @@ get_page() got_pages=$((${got_pages} + 1)) url="${baseurl}${bookid}&pg=${page}&jscmd=click3" - out "$(progress ${got_pages})" "${page}: ${url}&w=${pagewidth} TRY" + out "$(progress ${got_pages})" "${page}: ${url} TRY" # fetch urls # NB! signatures tied to cookie and ip - urls=$(wget ${optcommon} -U"${ua}" --load-cookies "${cookie}" -O- \ + urls=$(wget ${optcommon} -U"${ua}" --header "Cookie: ${cookie_str}" -O- \ "${url}" | tr '}' '\n' | \ sed -ne 's/^.*"src":"\(https:\/\/[^"]*\)".*$/\1/; /pg=/s/\\u0026/\&/gp') # fetch pages - for url in ${urls}; do + for url in ${urls}; do page=${url##*&pg=}; page=${page%%&*} # check again if page already downloaded, we usually get a few @@ -163,7 +157,7 @@ get_page() if [ ! -f "${bookid}/${page}.png" ]; then got_pages=$((${got_pages} + 1)) - wget ${optcommon} -U"${ua}" --load-cookies "${cookie}" \ + wget ${optcommon} -U"${ua}" --header "Cookie: ${cookie_str}" \ -O"${bookid}/${page}.png" "${url}&w=${pagewidth}" _return=$? @@ -245,7 +239,6 @@ make_pagelist() done } - # # MAIN # @@ -297,12 +290,8 @@ if [ ! -d "${bookid}" ]; then mkdir -- "${bookid}" || err 2 "cannot create dir ${bookid}" fi -cookie=`mktemp -t cookie` || err 2 'mktemp error' - -trap "rm ${cookie} 2>/dev/null; exit 1" 1 2 3 10 13 15 - get_retpages -echo "pages total/retrieved: ~${numpages}/${retpages}" +echo "pages available/fetched: ${numpages}+/${retpages}" if [ -z "${proxylist}" ]; then get_pages @@ -315,5 +304,3 @@ else echo done fi - -rm "${cookie}" 2>/dev/null