pymcurl 8.6.0.1

Description:

pymcurl 8.6.0.1

Manage outbound HTTP connections using Curl & CurlMulti
Description
mcurl is a Python wrapper for libcurl with a
high-level API that makes it easy to interact with the libcurl easy and multi
interfaces. It was originally created for the Px
proxy server which uses libcurl to handle upstream proxy authentication.
Usage
mcurl can be installed using pip:
pip install pymcurl

Binary packages are provided the following platforms:

aarch64-linux-gnu
aarch64-linux-musl
i686-linux-gnu
x86_64-linux-gnu
x86_64-linux-musl
x86_64-macos
x86_64-windows

mcurl leverages cffi to interface with libcurl
and all binary dependencies are sourced from binarybuilder.org.
auditwheel on Linux, delocate on MacOS and delvewheel on Windows are used to bundle
the shared libraries into the wheels.
Thanks to cffi
and Py_LIMITED_API,
these mcurl binaries should work on any Python from v3.2 onwards.
Easy interface
from mcurl import Curl

c = Curl('http://httpbin.org/get')
c.buffer()
ret = c.perform()
if ret == 0:
ret, resp = c.get_response()
headers = c.get_headers()
data = c.get_data()
print(f"Response: {resp}\n\n{headers}{data}")

Multi interface
from mcurl import Curl, MCurl

m = MCurl()

c1 = Curl('http://httpbin.org/get')
c1.buffer()
m.add(c1)

data = "test8192".encode("utf-8")
c2 = Curl('https://httpbin.org/post', 'POST')
c2.buffer(data=data)
c2.set_headers({"Content-Length": len(data)})
m.add(c2)

ret1 = m.do(c1)
ret2 = m.do(c2)

if ret1:
c1.get_response()
c1.get_headers()
c1.get_data()
print(f"Response: {c1.get_response()}\n\n{c1.get_headers()}{c1.get_data()}")
else:
print(f"Failed with error: {c1.errstr}")

if ret2:
c2.get_response()
c2.get_headers()
c2.get_data()
print(f"Response: {c2.get_response()}\n\n{c2.get_headers()}{c2.get_data()}")
else:
print(f"Failed with error: {c2.errstr}")

m.close()

libcurl API
The libcurl API can be directly accessed as is done
in mcurl if preferred.
from _libcurl_cffi import lib as libcurl
from _libcurl_cffi import ffi

url = "http://httpbin.org/get"
curl = ffi.new("char []", url.encode("utf-8"))

easy = libcurl.curl_easy_init()
libcurl.curl_easy_setopt(easy, libcurl.CURLOPT_URL, curl)
cerr = libcurl.curl_easy_perform(easy)

API reference
NAME
mcurl - Manage outbound HTTP connections using Curl & CurlMulti

CLASSES
builtins.object
Curl
MCurl

class Curl(builtins.object)
| Curl(url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
|
| Helper class to manage a curl easy instance
|
| Methods defined here:
|
| __del__(self)
| Destructor - clean up resources
|
| __init__(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
| Initialize curl instance
|
| method = GET, POST, PUT, CONNECT, etc.
| request_version = HTTP/1.0, HTTP/1.1, etc.
|
| bridge(self, client_rfile=None, client_wfile=None, client_hfile=None)
| Bridge curl reads/writes to sockets specified
|
| Reads POST/PATCH data from client_rfile
| Writes data back to client_wfile
| Writes headers back to client_hfile
|
| buffer(self, data=None)
| Setup buffers to bridge curl perform
|
| get_activesocket(self)
| Return active socket for this easy instance
|
| get_data(self, encoding='utf-8')
| Return data written by curl perform to buffer()
|
| encoding = "utf-8" by default, change or set to None if bytes preferred
|
| get_headers(self, encoding='utf-8')
| Return headers written by curl perform to buffer()
|
| encoding = "utf-8" by default, change or set to None if bytes preferred
|
| get_primary_ip(self)
| Return primary IP address of this easy instance
|
| get_response(self)
| Return response code of completed request
|
| perform(self)
| Perform the easy handle
|
| reset(self, url, method='GET', request_version='HTTP/1.1', connect_timeout=60)
| Reuse existing curl instance for another request
|
| set_auth(self, user, password=None, auth='ANY')
| Set proxy authentication info - call after set_proxy() to enable auth caching
|
| set_debug(self, enable=True)
| Enable debug output
| Call after set_proxy() and set_auth() to enable discovery and caching of proxy
| auth mechanism - libcurl does not provide an API to get this today - need to
| find it in sent header debug output
|
| set_follow(self, enable=True)
| Set curl to follow 3xx responses
|
| set_headers(self, xheaders)
| Set headers to send
|
| set_insecure(self, enable=True)
| Set curl to ignore SSL errors
|
| set_proxy(self, proxy, port=0, noproxy=None)
| Set proxy options - returns False if this proxy server has auth failures
|
| set_transfer_decoding(self, enable=False)
| Set curl to turn off transfer decoding - let client do it
|
| set_tunnel(self, tunnel=True)
| Set to tunnel through proxy if no proxy or proxy + auth
|
| set_useragent(self, useragent)
| Set user agent to send
|
| set_verbose(self, enable=True)
| Set verbose mode
|

class MCurl(builtins.object)
| MCurl(debug_print=None)
|
| Helper class to manage a curl multi instance
|
| Methods defined here:
|
| __init__(self, debug_print=None)
| Initialize multi interface
|
| add(self, curl: mcurl.Curl)
| Add a Curl handle to perform
|
| close(self)
| Stop any running transfers and close this multi handle
|
| do(self, curl: mcurl.Curl)
| Add a Curl handle and peform until completion
|
| remove(self, curl: mcurl.Curl)
| Remove a Curl handle once done
|
| select(self, curl: mcurl.Curl, client_sock, idle=30)
| Run select loop between client and curl
|
| setopt(self, option, value)
| Configure multi options
|
| stop(self, curl: mcurl.Curl)
| Stop a running curl handle and remove

FUNCTIONS
curl_version()

cvp2pystr(cvoidp)
Convert void * to Python string

debug_callback(easy, infotype, data, size, userp)
Prints out curl debug info and headers sent/received

dprint lambda x
# Debug shortcut

getauth(auth)
Return auth value for specified authentication string

Supported values can be found here: https://curl.se/libcurl/c/CURLOPT_HTTPAUTH.html

Skip the CURLAUTH_ portion in input - e.g. getauth("ANY")

To control which methods are available during proxy detection:
Prefix NO to avoid method - e.g. NONTLM => ANY - NTLM
Prefix SAFENO to avoid method - e.g. SAFENONTLM => ANYSAFE - NTLM
Prefix ONLY to support only that method - e.g ONLYNTLM => ONLY + NTLM

gethash(easy)
Return hash value for easy to allow usage as a dict key

header_callback(buffer, size, nitems, userdata)

multi_timer_callback(multi, timeout_ms, userp)

print_curl_version()
Display curl version information

py2cbool(pbool)
Convert Python bool to long

py2clong(plong)
Convert Python int to long

py2cstr(pstr)
Convert Python string to char *

py2custr(pstr)
Convert Python string to char *

read_callback(buffer, size, nitems, userdata)

sanitized(msg)
Hide user sensitive data from debug output

save_auth(curl, msg)
Find and cache proxy auth mechanism from headers sent by libcurl

save_upstream(curl, msg)
Find which server libcurl connected to - upstream proxy or target server

socket_callback(easy, sock_fd, ev_bitmask, userp, socketp)

sockopt_callback(clientp, sock_fd, purpose)

wa_callback(easy, infotype, data, size, userp)
curl debug callback to get info not provided by libcurl today
- proxy auth mechanism from sent headers
- upstream server connected to from curl info

write_callback(buffer, size, nitems, userdata)

yield_msgs(data, size)
Generator for curl debug messages


Building mcurl
mcurl is built using gcc on Linux, clang on MacOS and mingw-x64 on Windows. The
shared libraries are downloaded from binarybuilder.org
using jbb for Linux and Windows. MacOS packages the
libcurl binaries installed via brew.
build.sh can be used to build mcurl for all supported platforms including
Windows.