project/uclient.git
13 hours agouclient: fix http regression master
Felix Fietkau [Fri, 19 Apr 2024 18:04:54 +0000 (20:04 +0200)]
uclient: fix http regression

Signed-off-by: Felix Fietkau <nbd@nbd.name>
16 hours agouclient-http: use ustream_ssl without ustream_fd
Felix Fietkau [Fri, 19 Apr 2024 14:59:23 +0000 (16:59 +0200)]
uclient-http: use ustream_ssl without ustream_fd

Improves performance by avoiding double buffering

Signed-off-by: Felix Fietkau <nbd@nbd.name>
43 hours agouclient-fetch: add uclient / ustream-ssl log messages (enabled via -v)
Felix Fietkau [Thu, 18 Apr 2024 12:28:54 +0000 (14:28 +0200)]
uclient-fetch: add uclient / ustream-ssl log messages (enabled via -v)

Signed-off-by: Felix Fietkau <nbd@nbd.name>
43 hours agolib: add log_msg callback to get more detailed log messages
Felix Fietkau [Thu, 18 Apr 2024 12:28:09 +0000 (14:28 +0200)]
lib: add log_msg callback to get more detailed log messages

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2 weeks agoucode: add support for using a prototype for cb, pass it to callbacks
Felix Fietkau [Fri, 5 Apr 2024 17:03:54 +0000 (19:03 +0200)]
ucode: add support for using a prototype for cb, pass it to callbacks

This allows using the cb as dynamic data store for the request

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2 weeks agoucode: fix a few ucode binding issues
Felix Fietkau [Thu, 4 Apr 2024 18:45:43 +0000 (20:45 +0200)]
ucode: fix a few ucode binding issues

- fix resource refcounting bug by storing a resource pointer in the priv data
- handle writes from within the .request call

Signed-off-by: Felix Fietkau <nbd@nbd.name>
2 weeks agouclient: add function for getting the amount of pending read/write data
Felix Fietkau [Thu, 4 Apr 2024 18:45:16 +0000 (20:45 +0200)]
uclient: add function for getting the amount of pending read/write data

This can be used to throttle writes when doing large POST requests

Signed-off-by: Felix Fietkau <nbd@nbd.name>
3 weeks agoadd ucode binding
Felix Fietkau [Fri, 29 Mar 2024 13:09:52 +0000 (14:09 +0100)]
add ucode binding

Supports the full API of the C library and includes a simple example script

Signed-off-by: Felix Fietkau <nbd@nbd.name>
3 weeks agohttp: call ustream_poll if not enough read data is available
Felix Fietkau [Wed, 27 Mar 2024 21:29:38 +0000 (22:29 +0100)]
http: call ustream_poll if not enough read data is available

Avoids having to go through another uloop notification for the next read.

Signed-off-by: Felix Fietkau <nbd@nbd.name>
3 weeks agouclient: defer read notifications to uloop timer
Felix Fietkau [Fri, 29 Mar 2024 12:32:35 +0000 (13:32 +0100)]
uclient: defer read notifications to uloop timer

This gets rid of potentially harmful recursion when polling from within the
read callback

Signed-off-by: Felix Fietkau <nbd@nbd.name>
3 weeks agohttp: add helper function for checking redirect status
Felix Fietkau [Wed, 27 Mar 2024 10:52:49 +0000 (11:52 +0100)]
http: add helper function for checking redirect status

Signed-off-by: Felix Fietkau <nbd@nbd.name>
3 weeks agouclient: add helper function for getting ustream-ssl context/ops
Felix Fietkau [Wed, 27 Mar 2024 07:28:09 +0000 (08:28 +0100)]
uclient: add helper function for getting ustream-ssl context/ops

Signed-off-by: Felix Fietkau <nbd@nbd.name>
3 weeks agoadd strdupa macro for compatibility
Felix Fietkau [Tue, 26 Mar 2024 19:10:02 +0000 (20:10 +0100)]
add strdupa macro for compatibility

Signed-off-by: Felix Fietkau <nbd@nbd.name>
12 months agouclient: cancel state change timeout in uclient_disconnect()
Matthias Schiffer [Fri, 24 Feb 2023 21:25:07 +0000 (22:25 +0100)]
uclient: cancel state change timeout in uclient_disconnect()

When the uloop is ended right after the state change timeout has been
armed, the timeout will never be cancelled, without a way for the
libuclient user to clean up. When uclient_free() is then called on the
uclient context while the uloop is still live, the entry in uloop's timeout
list will be dangling, often resulting in a segfault when new timeouts are
added or the timeout list is cleaned up in uloop_done().

Fix this by cancelling the timeout in uclient_disconnect().

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
2 years agoci: improve wolfSSL test coverage
Petr Štetiar [Thu, 24 Feb 2022 08:57:19 +0000 (09:57 +0100)]
ci: improve wolfSSL test coverage

In order to prevent regressions like #9283, lets add build and runtime
testing of wolfSSL master and release branches into daily scheduled
pipeline.

References: https://github.com/openwrt/openwrt/issues/9283
References: https://github.com/wolfSSL/wolfssl/issues/4879
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2 years agotests: add certificate check against letsencrypt.org
Petr Štetiar [Thu, 24 Feb 2022 09:02:08 +0000 (10:02 +0100)]
tests: add certificate check against letsencrypt.org

In order to prevent regressions like #9283, lets add runtime check
against letsencrypt.org.

References: https://github.com/openwrt/openwrt/issues/9283
References: https://github.com/wolfSSL/wolfssl/issues/4879
Signed-off-by: Petr Štetiar <ynezz@true.cz>
2 years agouclient-http: set eof mark when content-length is 0
Youfu Zhang [Thu, 11 Mar 2021 12:58:33 +0000 (20:58 +0800)]
uclient-http: set eof mark when content-length is 0

or uclient-fetch will stall until timeout for 2XX (except 204) response
with content-length of 0

Signed-off-by: Youfu Zhang <zhangyoufu@gmail.com>
3 years agotests: fix help usage test for uclient built with sanitizer
Petr Štetiar [Sun, 4 Apr 2021 07:07:41 +0000 (09:07 +0200)]
tests: fix help usage test for uclient built with sanitizer

Commit 64e00d6db4c3 ("uclient-fetch: document missing options") has
changed output of usage help so adjust test accordingly.

Fixes: 64e00d6db4c3 ("uclient-fetch: document missing options")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agotests: fix help usage test
Petr Štetiar [Sun, 4 Apr 2021 07:07:41 +0000 (09:07 +0200)]
tests: fix help usage test

Commit 64e00d6db4c3 ("uclient-fetch: document missing options") has
changed output of usage help so adjust test accordingly.

Fixes: 64e00d6db4c3 ("uclient-fetch: document missing options")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-fetch: document missing options
Baptiste Jonglez [Wed, 30 Dec 2020 22:02:41 +0000 (23:02 +0100)]
uclient-fetch: document missing options

Missing ones were: --no-proxy, --quiet, --continue / -c

Also improved readability of several options.

Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org>
3 years agotests: fix possibly longer start of HTTP server
Petr Štetiar [Mon, 22 Mar 2021 20:16:51 +0000 (21:16 +0100)]
tests: fix possibly longer start of HTTP server

Seems like sometimes it can take some time to start HTTP server so let's
wait for the server becoming operating before executing test cases.

References: https://gitlab.com/openwrt/project/uclient/-/jobs/1118330555#L984
Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agoAdd initial GitLab CI support
Petr Štetiar [Tue, 8 Dec 2020 11:36:18 +0000 (12:36 +0100)]
Add initial GitLab CI support

Uses currently proof-of-concept openwrt-ci[1] in order to:

 * improve the quality of the codebase in various areas
 * decrease code review time and help merging contributions faster
 * get automagic feedback loop on various platforms and tools
   - out of tree build with OpenWrt SDK on following targets:
     * ath79-generic
     * imx6-generic
     * malta-be
     * mvebu-cortexa53
   - out of tree native build on x86/64 with GCC (versions 8, 9, 10) and Clang 10
   - out of tree native x86/64 static code analysis with cppcheck and
     scan-build from Clang 10

1. https://gitlab.com/ynezz/openwrt-ci/

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-fetch: wolfSSL: fix certificate validation
Petr Štetiar [Thu, 10 Dec 2020 09:58:55 +0000 (10:58 +0100)]
uclient-fetch: wolfSSL: fix certificate validation

Currently wolfSSL doesn't validate any certificates, quoting from
README:

 wolfSSL takes a different approach to certificate verification than
 OpenSSL does. The default policy for the client is to verify the server,
 this means that if you don't load CAs to verify the server you'll get a
 connect error, no signer error to confirm failure (-188).

 If you want to mimic OpenSSL behavior of having SSL_connect succeed even if
 verifying the server fails and reducing security you can do this by calling:

 wolfSSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, 0);

 before calling wolfSSL_new();. Though it's not recommended.

wolfSSL simply behaves differently then OpenSSL so once you set
SSL_VERIFY_NONE wolfSSL doesn't care about the certificates anymore so
every call to SSL_get_verify_result() is going to succeed (returns
X509_V_OK) even for invalid certificates and current OpenSSL based post
connection verification logic thus doesn't work.

So in order to get the validation working we need to use SSL_VERIFY_PEER
for wolfSSL by default and allow disabling it explicitly by new
`context_set_require_validation()` call.

Fixes: FS#3465
Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-fetch: init_ca_cert: fix memory leak
Petr Štetiar [Tue, 8 Dec 2020 17:54:57 +0000 (18:54 +0100)]
uclient-fetch: init_ca_cert: fix memory leak

Fixes following memory leak:

 $ valgrind --quiet --leak-check=full uclient-fetch -q http://127.0.0.1:1922/does-not-exist

  51 (16 direct, 35 indirect) bytes in 1 blocks are definitely lost in loss record 4 of 9
   at 0x4C31A3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
   by 0x4C33D84: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
   by 0x553C7DF: glob_in_dir (glob.c:1410)
   by 0x553D5E8: glob@@GLIBC_2.27 (glob.c:1097)
   by 0x401D62: init_ca_cert (uclient-fetch.c:503)
   by 0x401D62: main (uclient-fetch.c:741)

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agocmake: enable extra compiler checks
Petr Štetiar [Tue, 8 Dec 2020 16:23:36 +0000 (17:23 +0100)]
cmake: enable extra compiler checks

Let's enforce additional automatic checks enforced by the compiler in
order to catch possible errors during compilation.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-http: fix extra compiler warnings on mips_24kc and cortex-a9+neon
Petr Štetiar [Tue, 8 Dec 2020 17:13:03 +0000 (18:13 +0100)]
uclient-http: fix extra compiler warnings on mips_24kc and cortex-a9+neon

Fixes following warnings as reported on 32-bit platforms
toolchain-mips_24kc_gcc-8.4.0_musl and toolchain-arm_cortex-a9+neon_gcc-8.4.0_musl_eabi:

 uclient-http.c:1111:10: error: comparison of integer expressions of different signedness: 'unsigned int' and 'int' [-Werror=sign-compare]
   if (len > data_end - data)
           ^
 uclient-http.c:1115:11: error: comparison of integer expressions of different signedness: 'unsigned int' and 'long int' [-Werror=sign-compare]
    if (len > uh->read_chunked)
            ^
 uclient-http.c:1120:11: error: comparison of integer expressions of different signedness: 'unsigned int' and 'long int' [-Werror=sign-compare]
    if (len > uh->content_length)
            ^

References: https://gitlab.com/ynezz/openwrt-uclient/-/pipelines/226912126
Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-fetch: fix potential memory leaks
Petr Štetiar [Tue, 8 Dec 2020 16:45:19 +0000 (17:45 +0100)]
uclient-fetch: fix potential memory leaks

Fixes following issue reported by clang-12 static analyzer:

 uclient-fetch.c:612:25: warning: Potential leak of memory pointed to by 'username' [unix.Malloc]
                                 memset(optarg, '*', strlen(optarg));
                                                     ^~~~~~~~~~~~~~
 uclient-fetch.c:618:25: warning: Potential leak of memory pointed to by 'password' [unix.Malloc]
                                 memset(optarg, '*', strlen(optarg));
                                                     ^~~~~~~~~~~~~~

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient: fix initialized but never read variable
Petr Štetiar [Tue, 8 Dec 2020 16:41:44 +0000 (17:41 +0100)]
uclient: fix initialized but never read variable

Fixes following issue reported by clang-12 static analyzer:

 uclient.c:290:22: warning: Value stored to 'url' during its initialization is never read [deadcode.DeadStores]
         struct uclient_url *url = cl->url;
                             ^~~   ~~~~~~~

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-fetch: fix statement may fallt hrough
Petr Štetiar [Tue, 8 Dec 2020 16:37:52 +0000 (17:37 +0100)]
uclient-fetch: fix statement may fallt hrough

Fixes following issue reported by clang-12 static analyzer:

 uclient-fetch.c:228:6: error: this statement may fall through [-Werror=implicit-fallthrough=]
    if (sscanf(blobmsg_get_string(tb[H_RANGE]),
       ^
 uclient-fetch.c:236:2: note: here
   case 204:
   ^~~~

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-http: fix freeing of stack allocated memory
Petr Štetiar [Tue, 8 Dec 2020 16:34:50 +0000 (17:34 +0100)]
uclient-http: fix freeing of stack allocated memory

Fixes following issue reported by clang-12 static analyzer:

 uclient-http.c:568:2: warning: Memory allocated by alloca() should not be deallocated [unix.Malloc]
         free(buf_orig);
         ^~~~~~~~~~~~~~

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agoFix extra compiler warnings
Petr Štetiar [Tue, 8 Dec 2020 16:29:57 +0000 (17:29 +0100)]
Fix extra compiler warnings

Fixes following -Wextra compiler warnings:

 uclient.c:195:16: error: comparison of integers of different signs: 'int' and 'unsigned long' [-Werror,-Wsign-compare]
         for (i = 0; i < ARRAY_SIZE(backends); i++) {
                     ~ ^ ~~~~~~~~~~~~~~~~~~~~

 uclient-http.c:619:2: error: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'int' [-Werror,-Wsign-compare]
         blobmsg_for_each_attr(cur, uh->headers.head, rem)
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 uclient-http.c:619:2: error: comparison of integers of different signs: 'int' and 'unsigned long' [-Werror,-Wsign-compare]
         blobmsg_for_each_attr(cur, uh->headers.head, rem)
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 uclient-http.c:993:16: error: comparison of integers of different signs: 'int' and 'unsigned long' [-Werror,-Wsign-compare]
         for (i = 0; i < ARRAY_SIZE(request_types); i++) {
                     ~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~

 uclient.c:195:16: error: comparison of integers of different signs: 'int' and 'unsigned long' [-Werror,-Wsign-compare]
         for (i = 0; i < ARRAY_SIZE(backends); i++) {
                     ~ ^ ~~~~~~~~~~~~~~~~~~~~

 uclient-http.c:619:2: error: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'int' [-Werror,-Wsign-compare]
         blobmsg_for_each_attr(cur, uh->headers.head, rem)
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 uclient-http.c:619:2: error: comparison of integers of different signs: 'int' and 'unsigned long' [-Werror,-Wsign-compare]
         blobmsg_for_each_attr(cur, uh->headers.head, rem)
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 uclient-http.c:993:16: error: comparison of integers of different signs: 'int' and 'unsigned long' [-Werror,-Wsign-compare]
         for (i = 0; i < ARRAY_SIZE(request_types); i++) {
                     ~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~

 uclient-fetch.c:551:67: error: missing field 'flag' initializer [-Werror,-Wmissing-field-initializers]
         [L_NO_CHECK_CERTIFICATE] = { "no-check-certificate", no_argument },

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agoAdd basic cram based unit tests
Petr Štetiar [Tue, 8 Dec 2020 11:29:59 +0000 (12:29 +0100)]
Add basic cram based unit tests

For the start just basic uclient-fetch functionality coverage.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agocmake: fix building out of the tree
Petr Štetiar [Tue, 8 Dec 2020 09:06:29 +0000 (10:06 +0100)]
cmake: fix building out of the tree

When building out of the tree, linker is unable to find the ubox library
so fix it by using find_library CMake command.

Signed-off-by: Petr Štetiar <ynezz@true.cz>
3 years agouclient-fetch: add option to read POST data from file
Gio [Fri, 12 Jun 2020 10:59:52 +0000 (12:59 +0200)]
uclient-fetch: add option to read POST data from file

Passing post data in command line is convenient but has limited size,
and may become tricky to correctly escape passed data especially in
scripts.
This patch add the option --post-file so the data to post can be read
from a file.
Tested on x86/64.

Acked-by: Jo-Philipp Wich <jo@mein.io>
Signed-off-by: Gioacchino Mazzurco <gio@eigenlab.org>
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
4 years agouclient-fetch: support specifying advertised TLS ciphers
Jo-Philipp Wich [Sat, 15 Feb 2020 21:39:30 +0000 (22:39 +0100)]
uclient-fetch: support specifying advertised TLS ciphers

Introduce a new `--ciphers` option which allows specifying a colon separated
list of usable TLS ciphers.

Depending on the underlying ustream-ssl provider, the list either follows
OpenSSL's cipher string format or, in case of mbedTLS, is a simple colon
separated cipher whitelist.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
4 years agouclient: Add string error function
Daniel Danzberger [Thu, 5 Dec 2019 16:51:58 +0000 (17:51 +0100)]
uclient: Add string error function

This add's the uclient_strerror function, which resolves and error code
to a string message.

Signed-off-by: Daniel Danzberger <daniel@dd-wrt.com>
4 years agouclient-http: set data_eof when content-length is 0
Yousong Zhou [Mon, 27 May 2019 02:31:57 +0000 (02:31 +0000)]
uclient-http: set data_eof when content-length is 0

Otherwise uclient-fetch can report "Connection reset prematurely"

Fixes FS#2222

Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
Acked-by: Petr Štetiar <ynezz@true.cz>
5 years agouclient-http: properly handle HTTP redirects via proxy connections
Jo-Philipp Wich [Sat, 24 Nov 2018 18:54:07 +0000 (19:54 +0100)]
uclient-http: properly handle HTTP redirects via proxy connections

When uclient connects via an HTTP proxy connection we need to swap the
contexts .proxy_url and not the .url property (which is the connection
URL to the proxy server itself).

This fixes uclient erroring out in a redirect loop when connecting via
a proxy connection.

Ref: https://forum.openwrt.org/t/opkg-update-fails/25813
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
5 years agouclient-http: Close ustream file handle only if allocated
Tobias Schramm [Fri, 3 Aug 2018 21:39:21 +0000 (23:39 +0200)]
uclient-http: Close ustream file handle only if allocated

Since the connection setup in uclient_do_connect can fail before
ustream_init_fd is called we must check the fd was actually allocated
before closing it, else we would close STDIN.

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
6 years agouclient-utils: Handle memory allocation failure for url file name
Tobias Schramm [Tue, 20 Feb 2018 10:26:31 +0000 (11:26 +0100)]
uclient-utils: Handle memory allocation failure for url file name

Add null pointer check to allocation of url filename

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
6 years agouclient-http: Implement error handling for header-sending
Tobias Schramm [Tue, 20 Feb 2018 10:26:29 +0000 (11:26 +0100)]
uclient-http: Implement error handling for header-sending

Add some error handling for errors during assembly of headers

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
6 years agouclient: Handle memory allocation failure for url
Tobias Schramm [Sun, 18 Feb 2018 12:46:10 +0000 (13:46 +0100)]
uclient: Handle memory allocation failure for url

Check ptr to url after allocation to prevent null pointer
dereference

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
6 years agouclient-http: Handle memory allocation failure
Tobias Schramm [Sun, 18 Feb 2018 12:46:08 +0000 (13:46 +0100)]
uclient-http: Handle memory allocation failure

Add null pointer check to allocation of uclient_http struct

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
6 years agouclient-http: auth digest: Handle multiple possible memory allocation failures
Tobias Schramm [Sun, 18 Feb 2018 12:46:06 +0000 (13:46 +0100)]
uclient-http: auth digest: Handle multiple possible memory allocation failures

Add null pointer checks to allocation of buffers for authentication parameters

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
6 years agouclient-http: basic auth: Handle memory allocation failure
Tobias Schramm [Sun, 18 Feb 2018 12:46:05 +0000 (13:46 +0100)]
uclient-http: basic auth: Handle memory allocation failure

Allocation of the base64 buffer might fail, resulting in a null
ptr being passed to base64_encode as a target buffer

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
6 years agouclient-fetch: Check for nullpointer returned by uclient_get_url_filename
Tobias Schramm [Sun, 18 Feb 2018 12:46:04 +0000 (13:46 +0100)]
uclient-fetch: Check for nullpointer returned by uclient_get_url_filename

uclient_get_url_filename may return a null ptr resulting in a null
ptr dereference by fprintf and open

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
6 years agouclient-fetch: use package name pattern in message for missing SSL library
Michael Heimpold [Thu, 25 Jan 2018 20:53:45 +0000 (21:53 +0100)]
uclient-fetch: use package name pattern in message for missing SSL library

When faced with this error message, I did not directly know what
package(s) to install. I first grepped the package list for packages
named libustream-ssl... found nothing, and finally realized, that here
only the library name was mentioned.

But when we mention the certificate packages, why not also directly
mention the SSL/TLS packages with a pattern. That will save one step
next time when stumbling here - at least for me ;-)

Signed-off-by: Michael Heimpold <mhei@heimpold.de>
6 years agouclient-fetch: fix overloading of output_file variable
Felix Fietkau [Thu, 2 Nov 2017 20:58:42 +0000 (21:58 +0100)]
uclient-fetch: fix overloading of output_file variable

When uclient-fetch is called with multiple URL's, it derives the
first filename from the URL. It then sets the global output_file
variable, causing a use-after-free bug on an attempt to use it as output
file for the next file.

Fix this by avoiding the overwrite entirely by only setting a local
variable

Reported-by: Philip Prindeville <philipp@redfish-solutions.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
6 years agouclient-http: fix Host: header for literal IPv6 addresses
Matthias Schiffer [Tue, 5 Sep 2017 18:02:37 +0000 (20:02 +0200)]
uclient-http: fix Host: header for literal IPv6 addresses

For literal IPv6 addresses, the host header must have the form

  Host: [...]

including the square brackets, as it may also contain a port. Some
webservers ignore the missing brackets, while others will return error 400.

IPv6 addresses are determined by searching for ':' characters in the host,
as neither IPv4 addresses nor DNS names can contain colons. An alternative
would be to add a flag to the uclient_url struct; but as this struct is
exposed in public headers, such a change could be considered an ABI change,
making a backport difficult.

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Acked-by: Jo-Philipp Wich <jo@mein.io>
6 years agouclient-fetch: read_data_cb: fix a potential buffer overflow
Alexander Couzens [Sun, 11 Jun 2017 11:01:06 +0000 (13:01 +0200)]
uclient-fetch: read_data_cb: fix a potential buffer overflow

When a backend->read() is not implement or returns -1,
This -1 is interpreted as unsigned int resulting
in an access far behind the buffer.

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
7 years agoremove obsolete mac os x /opt/local include/library search path
Felix Fietkau [Fri, 9 Dec 2016 12:06:59 +0000 (13:06 +0100)]
remove obsolete mac os x /opt/local include/library search path

Where necessary, this can easily be provided through CFLAGS/LDFLAGS

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years agoFix unused results warnings
Florian Fainelli [Thu, 8 Dec 2016 03:04:45 +0000 (19:04 -0800)]
Fix unused results warnings

Fixes:

uclient-http.c:385:8: error: ignoring return value of 'fread', declared with attribute warn_unused_result [-Werror=unused-result]
   fread(&val, sizeof(val), 1, f);
        ^

uclient-fetch.c: In function 'main':
uclient-fetch.c:664:12: error: ignoring return value of 'asprintf', declared with attribute warn_unused_result [-Werror=unused-result]
    asprintf(&auth_str, "%s:%s", username, password);
            ^
uclient-fetch.c: In function 'read_data_cb':
uclient-fetch.c:269:9: error: ignoring return value of 'write', declared with attribute warn_unused_result [-Werror=unused-result]
    write(output_fd, buf, len);

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
7 years agouclient-http: send correct "Host:" header if port is set
Alexander Couzens [Wed, 30 Nov 2016 17:49:10 +0000 (18:49 +0100)]
uclient-http: send correct "Host:" header if port is set

When connecting to a website with a special port, uclient-fetch connects
to the correct port, but is sending an incorrect Host: header without
the corresponding port.

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
7 years agoFix printf format in previous commit.
Jo-Philipp Wich [Sat, 30 Jul 2016 13:35:10 +0000 (15:35 +0200)]
Fix printf format in previous commit.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years agoMake missing SSL support message more explicit.
Jo-Philipp Wich [Sat, 30 Jul 2016 13:33:52 +0000 (15:33 +0200)]
Make missing SSL support message more explicit.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>
7 years agohttp: add support for relative location on redirect
Felix Fietkau [Tue, 5 Jul 2016 10:39:01 +0000 (12:39 +0200)]
http: add support for relative location on redirect

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years agouclient-fetch: add support for --quiet
Felix Fietkau [Tue, 5 Jul 2016 09:19:38 +0000 (11:19 +0200)]
uclient-fetch: add support for --quiet

Signed-off-by: Felix Fietkau <nbd@nbd.name>
7 years agocmake: Find libubox/ustream-ssl.h header file
Florian Fainelli [Fri, 1 Jul 2016 23:03:47 +0000 (16:03 -0700)]
cmake: Find libubox/ustream-ssl.h header file

Add a CMake FIND_PATH and INCLUDE_DIRECTORIES searching for
libubox/ustream-ssl.h.  Some external toolchains which do not include
standard locations would fail to find the header otherwise.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
7 years agohttp: allow sending message body for DELETE request
Rafał Miłecki [Thu, 16 Jun 2016 12:10:05 +0000 (14:10 +0200)]
http: allow sending message body for DELETE request

Sending entity within DELETE is not forbidden by RFC 7231, see section
4.3.5. DELETE:
> A payload within a DELETE request message has no defined semantics;
> sending a payload body on a DELETE request might cause some existing
> implementations to reject the request.

Signed-off-by: Rafał Miłecki <zajec5@gmail.com>
8 years agouclient-fetch: fix inconsistencies in help text (thx, Hannu Nyman)
Felix Fietkau [Mon, 8 Feb 2016 18:03:57 +0000 (19:03 +0100)]
uclient-fetch: fix inconsistencies in help text (thx, Hannu Nyman)

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: use usock_inet_timeout()
Felix Fietkau [Thu, 28 Jan 2016 11:19:54 +0000 (12:19 +0100)]
http: use usock_inet_timeout()

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: add support for specifying ipv4/ipv6 preference
Felix Fietkau [Sun, 24 Jan 2016 23:18:33 +0000 (00:18 +0100)]
http: add support for specifying ipv4/ipv6 preference

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: truncate output files (unless resuming)
Felix Fietkau [Sat, 23 Jan 2016 19:59:45 +0000 (20:59 +0100)]
uclient-fetch: truncate output files (unless resuming)

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: assume data EOF if the connection terminates
Felix Fietkau [Sat, 23 Jan 2016 18:59:25 +0000 (19:59 +0100)]
http: assume data EOF if the connection terminates

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: fix progress bar initialization with -O-
Felix Fietkau [Sat, 23 Jan 2016 18:52:58 +0000 (19:52 +0100)]
uclient-fetch: fix progress bar initialization with -O-

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: fix processing server http data separated by \n instead of \r\n
Felix Fietkau [Sat, 23 Jan 2016 18:49:55 +0000 (19:49 +0100)]
http: fix processing server http data separated by \n instead of \r\n

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: add proper error handling to uclient_http_redirect()
Felix Fietkau [Sat, 23 Jan 2016 00:23:00 +0000 (01:23 +0100)]
http: add proper error handling to uclient_http_redirect()

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: allow overwriting files if -O was used
Felix Fietkau [Thu, 21 Jan 2016 15:58:26 +0000 (16:58 +0100)]
uclient-fetch: allow overwriting files if -O was used

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: fix short command line option for timeout
Felix Fietkau [Wed, 20 Jan 2016 09:52:40 +0000 (10:52 +0100)]
uclient-fetch: fix short command line option for timeout

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: fix connection close handling on auth
Felix Fietkau [Wed, 20 Jan 2016 09:51:58 +0000 (10:51 +0100)]
http: fix connection close handling on auth

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add missing newline after final progress bar update
Felix Fietkau [Tue, 19 Jan 2016 21:59:02 +0000 (22:59 +0100)]
uclient-fetch: add missing newline after final progress bar update

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agocore: fix unused but set variable
Felix Fietkau [Sun, 17 Jan 2016 12:39:51 +0000 (13:39 +0100)]
core: fix unused but set variable

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agocore: make __uclient_get_url static
Felix Fietkau [Sun, 17 Jan 2016 12:39:23 +0000 (13:39 +0100)]
core: make __uclient_get_url static

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add progress bar support
Felix Fietkau [Sun, 17 Jan 2016 12:37:59 +0000 (13:37 +0100)]
uclient-fetch: add progress bar support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add support for setting output directory
Felix Fietkau [Sat, 16 Jan 2016 22:39:10 +0000 (23:39 +0100)]
uclient-fetch: add support for setting output directory

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-http: use blocking connect, switch to non-blocking later. fixes ipv6 -> ipv4...
Felix Fietkau [Sat, 16 Jan 2016 22:29:51 +0000 (23:29 +0100)]
uclient-http: use blocking connect, switch to non-blocking later. fixes ipv6 -> ipv4 fallback

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: detect connect failure
Felix Fietkau [Sat, 16 Jan 2016 22:26:32 +0000 (23:26 +0100)]
http: detect connect failure

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: ignore SIGPIPE
Felix Fietkau [Sat, 16 Jan 2016 22:25:38 +0000 (23:25 +0100)]
uclient-fetch: ignore SIGPIPE

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add proxy support
Felix Fietkau [Sat, 16 Jan 2016 22:08:55 +0000 (23:08 +0100)]
uclient-fetch: add proxy support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: add proxy support
Felix Fietkau [Sat, 16 Jan 2016 22:08:46 +0000 (23:08 +0100)]
http: add proxy support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agocore: add proxy support
Felix Fietkau [Sat, 16 Jan 2016 22:08:31 +0000 (23:08 +0100)]
core: add proxy support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agocore: fix memory leak if url change fails
Felix Fietkau [Sat, 16 Jan 2016 19:13:36 +0000 (20:13 +0100)]
core: fix memory leak if url change fails

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-http: make username/password variables local
Felix Fietkau [Sat, 16 Jan 2016 18:46:33 +0000 (19:46 +0100)]
uclient-http: make username/password variables local

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add support for resuming downloads
Felix Fietkau [Sat, 16 Jan 2016 18:34:22 +0000 (19:34 +0100)]
uclient-fetch: add support for resuming downloads

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: clean up output file selection
Felix Fietkau [Sat, 16 Jan 2016 18:18:38 +0000 (19:18 +0100)]
uclient-fetch: clean up output file selection

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: allow the header_done callback to reset the client state
Felix Fietkau [Sat, 16 Jan 2016 17:50:22 +0000 (18:50 +0100)]
http: allow the header_done callback to reset the client state

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add support for --timeout
Felix Fietkau [Sat, 16 Jan 2016 12:08:45 +0000 (13:08 +0100)]
uclient-fetch: add support for --timeout

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add support for --spider
Felix Fietkau [Sat, 16 Jan 2016 11:55:36 +0000 (12:55 +0100)]
uclient-fetch: add support for --spider

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add support for --post-data
Felix Fietkau [Sat, 16 Jan 2016 11:50:11 +0000 (12:50 +0100)]
uclient-fetch: add support for --post-data

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: only add default CA entries if no certificate was supplied and verify...
Felix Fietkau [Sat, 16 Jan 2016 11:22:00 +0000 (12:22 +0100)]
uclient-fetch: only add default CA entries if no certificate was supplied and verify was requested

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add support for setting the user agent
Felix Fietkau [Sat, 16 Jan 2016 11:20:09 +0000 (12:20 +0100)]
uclient-fetch: add support for setting the user agent

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add support for fetching multiple urls
Felix Fietkau [Sat, 16 Jan 2016 10:43:40 +0000 (11:43 +0100)]
uclient-fetch: add support for fetching multiple urls

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: add user/password to command line help
Felix Fietkau [Sat, 16 Jan 2016 10:01:58 +0000 (11:01 +0100)]
uclient-fetch: add user/password to command line help

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: load CA certificates
Felix Fietkau [Sat, 16 Jan 2016 00:17:33 +0000 (01:17 +0100)]
uclient-fetch: load CA certificates

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agohttp: get remote address from usock instead of using getpeername
Felix Fietkau [Fri, 15 Jan 2016 20:30:09 +0000 (21:30 +0100)]
http: get remote address from usock instead of using getpeername

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
8 years agouclient-fetch: set server_name of the ssl context to support SNI
Alexander Couzens [Thu, 6 Aug 2015 11:02:47 +0000 (13:02 +0200)]
uclient-fetch: set server_name of the ssl context to support SNI

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
9 years agouclient-http: end custom header lines with CRLF
Felix Fietkau [Tue, 14 Apr 2015 21:04:32 +0000 (23:04 +0200)]
uclient-http: end custom header lines with CRLF

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
9 years agouclient-http: allow .header_done() callback to reset the connection
Felix Fietkau [Tue, 14 Apr 2015 21:01:53 +0000 (23:01 +0200)]
uclient-http: allow .header_done() callback to reset the connection

Signed-off-by: Felix Fietkau <nbd@openwrt.org>
9 years agoallow sending requests with DELETE method
Rafał Miłecki [Thu, 22 Jan 2015 10:45:13 +0000 (11:45 +0100)]
allow sending requests with DELETE method

Signed-off-by: Rafał Miłecki <zajec5@gmail.com>
9 years agouse const for char buffer in uclient_write
Rafał Miłecki [Wed, 21 Jan 2015 21:48:35 +0000 (22:48 +0100)]
use const for char buffer in uclient_write

We are not supposed to modify it and ustream accepts const already.

Signed-off-by: Rafał Miłecki <zajec5@gmail.com>