Learning Pingora 05 - Connect with TLS

Published: at 22:11

在最基本的 L4 连接建立完成之后，如果有需要，Pingora 就可以在此基础之开始建立 TLS 连接了。TLS 建立有两种选择：openssl 和 boringssl。Pingora 分别为二者包装了 pingora-openssl 和 pingora-boringssl 两个库。

在本文中，我们以 OpenSSL 为核心去描绘基本的连接建立流程，因此与 BoringSSL 相关的代码将会折叠。

事先声明，这篇文章整体会比较水，因为 TLS 客户端的实现其实也没什么可说的，而且基本都是一看就懂。所以本文也不会讲解太多。

此外，对 Rustls 的支持在 #29 中讨论。

ToC

connect

TLS 部分的 connect 基本就是根据 peer 的情况做一些参数上的调整，包括但不限于 ca、cert、可用的 curve、second keyshare 相关的检查与设置。在这些都配置完之后，就进入了 handshake 的流程。根据 connection_timeout 的配置，handshake 会有一个可选的超时时间。

pingora-core/src/tls/connectors/tls.rs
pub(crate) async fn connect<T, P>(
    stream: T,
    peer: &P,
    alpn_override: Option<ALPN>,
    tls_ctx: &SslConnector,
) -> Result<SslStream<T>>
where
    T: IO,
    P: Peer + Send + Sync,
{
    let mut ssl_conf = tls_ctx.configure().unwrap();
3 collapsed lines


    ssl_set_renegotiate_mode_freely(&mut ssl_conf);


    // Set up CA/verify cert store
    // TODO: store X509Store in the peer directly
    if let Some(ca_list) = peer.get_ca() {
        let mut store_builder = X509StoreBuilder::new().unwrap();
        for ca in &***ca_list {
            store_builder.add_cert(ca.clone()).unwrap();
        }
        ssl_set_verify_cert_store(&mut ssl_conf, &store_builder.build())
            .or_err(InternalError, "failed to load cert store")?;
    }


    // Set up client cert/key
    if let Some(key_pair) = peer.get_client_cert_key() {
        debug!("setting client cert and key");
        ssl_use_certificate(&mut ssl_conf, key_pair.leaf())
            .or_err(InternalError, "invalid client cert")?;
        ssl_use_private_key(&mut ssl_conf, key_pair.key())
            .or_err(InternalError, "invalid client key")?;


        let intermediates = key_pair.intermediates();
        if !intermediates.is_empty() {
            debug!("adding intermediate certificates for mTLS chain");
            for int in intermediates {
                ssl_add_chain_cert(&mut ssl_conf, int)
                    .or_err(InternalError, "invalid intermediate client cert")?;
            }
        }
    }


    if let Some(curve) = peer.get_peer_options().and_then(|o| o.curves) {
        ssl_set_groups_list(&mut ssl_conf, curve).or_err(InternalError, "invalid curves")?;
    }
6 collapsed lines


    // second_keyshare is default true
    if !peer.get_peer_options().map_or(true, |o| o.second_keyshare) {
        ssl_use_second_key_share(&mut ssl_conf, false);
    }


    // disable verification if sni does not exist
    // XXX: verify on empty string cause null string seg fault
    if peer.sni().is_empty() {
        ssl_conf.set_use_server_name_indication(false);
        /* NOTE: technically we can still verify who signs the cert but turn it off to be
        consistent with nginx's behavior */
        ssl_conf.set_verify(SslVerifyMode::NONE);
    } else if peer.verify_cert() {
        if peer.verify_hostname() {
            let verify_param = ssl_conf.param_mut();
            add_host(verify_param, peer.sni()).or_err(InternalError, "failed to add host")?;
            // if sni had underscores in leftmost label replace and add
            if let Some(sni_s) = replace_leftmost_underscore(peer.sni()) {
                add_host(verify_param, sni_s.as_ref()).unwrap();
            }
            if let Some(alt_cn) = peer.alternative_cn() {
                if !alt_cn.is_empty() {
                    add_host(verify_param, alt_cn).unwrap();
                    // if alt_cn had underscores in leftmost label replace and add
                    if let Some(alt_cn_s) = replace_leftmost_underscore(alt_cn) {
                        add_host(verify_param, alt_cn_s.as_ref()).unwrap();
                    }
                }
            }
        }
        ssl_conf.set_verify(SslVerifyMode::PEER);
    } else {
        ssl_conf.set_verify(SslVerifyMode::NONE);
    }


    /*
       We always set set_verify_hostname(false) here because:
        - verify case.)  otherwise ssl.connect calls X509_VERIFY_PARAM_set1_host
                         which overrides the names added by add_host. Verify is
                         essentially on as long as the names are added.
        - off case.)    the non verify hostname case should have it disabled
    */
    ssl_conf.set_verify_hostname(false);


    if let Some(alpn) = alpn_override.as_ref().or(peer.get_alpn()) {
        ssl_conf.set_alpn_protos(alpn.to_wire_preference()).unwrap();
    }


    clear_error_stack();
    let connect_future = handshake(ssl_conf, peer.sni(), stream);


    match peer.connection_timeout() {
        Some(t) => match pingora_timeout::timeout(t, connect_future).await {
            Ok(res) => res,
            Err(_) => Error::e_explain(
                ConnectTimedout,
                format!("connecting to server {}, timeout {:?}", peer, t),
            ),
        },
        None => connect_future.await,
    }
}

handshake

和 connect 相比，handshake 能讲的其实更少了。主要的内容就是两个：36 行的 SslSteram::new 和 38 行的 connect。

其中，SslStream 是 pingora 对不同 SSL 实现的抽象这个我们接下来马上会讲到；而 connect 则是直接调用了各种 SSL 实现的 connect 方法，比如 openssl 就是调用了 tokio-openssl 的 SslStream::connect。

其他本身大部分都是在做错误处理，这里就不多赘述了。

pingora-core/src/protocols/ssl/client.rs
/// Perform the TLS handshake for the given connection with the given configuration
pub async fn handshake<S: IO>(
    conn_config: ConnectConfiguration,
    domain: &str,
    io: S,
) -> Result<SslStream<S>> {
    let ssl = conn_config
        .into_ssl(domain)
        .explain_err(TLSHandshakeFailure, |e| format!("ssl config error: {e}"))?;
    let mut stream = SslStream::new(ssl, io)
        .explain_err(TLSHandshakeFailure, |e| format!("ssl stream error: {e}"))?;
    let handshake_result = stream.connect().await;
    match handshake_result {
33 collapsed lines
        Ok(()) => Ok(stream),
        Err(e) => {
            let context = format!("TLS connect() failed: {e}, SNI: {domain}");
            match e.code() {
                ssl::ErrorCode::SSL => {
                    // Unify the return type of `verify_result` for openssl
                    #[cfg(not(feature = "boringssl"))]
                    fn verify_result<S>(stream: SslStream<S>) -> Result<(), i32> {
                        match stream.ssl().verify_result().as_raw() {
                            crate::tls::ssl_sys::X509_V_OK => Ok(()),
                            e => Err(e),
                        }
                    }


                    // Unify the return type of `verify_result` for boringssl
                    #[cfg(feature = "boringssl")]
                    fn verify_result<S>(stream: SslStream<S>) -> Result<(), i32> {
                        stream.ssl().verify_result().map_err(|e| e.as_raw())
                    }


                    match verify_result(stream) {
                        Ok(()) => Error::e_explain(TLSHandshakeFailure, context),
                        // X509_V_ERR_INVALID_CALL in case verify result was never set
                        Err(X509_V_ERR_INVALID_CALL) => {
                            Error::e_explain(TLSHandshakeFailure, context)
                        }
                        _ => Error::e_explain(InvalidCert, context),
                    }
                }
                /* likely network error, but still mark as TLS error */
                _ => Error::e_explain(TLSHandshakeFailure, context),
            }
        }
    }
}

SslStream

先来看定义吧。这个 struct 里定义了：

pingora-core/src/protocols/ssl/mod.rs
/// The TLS connection
#[derive(Debug)]
pub struct SslStream<T> {
    ssl: InnerSsl<T>,
    digest: Option<Arc<SslDigest>>,
    timing: TimingDigest,
}

来看主要的部分。可以看到，SslStream 对 T 的要求是 AsyncRead + AsyncWrite + Unpin。由于 SslStream 本身需要 Async Readable/Writable，所以自然其内部承载的流也需要具备该特性。初次之外就都是对 self.ssl 的一般包装，看看就好（

pingora-core/src/protocols/ssl/mod.rs
impl<T> SslStream<T>
where
    T: AsyncRead + AsyncWrite + std::marker::Unpin,
{
    /// Create a new TLS connection from the given `stream`
    ///
    /// The caller needs to perform [`Self::connect()`] or [`Self::accept()`] to perform TLS
    /// handshake after.
    pub fn new(ssl: ssl::Ssl, stream: T) -> Result<Self> {
        let ssl = InnerSsl::new(ssl, stream)
            .explain_err(TLSHandshakeFailure, |e| format!("ssl stream error: {e}"))?;


        Ok(SslStream {
            ssl,
            digest: None,
            timing: Default::default(),
        })
    }


    /// Connect to the remote TLS server as a client
    pub async fn connect(&mut self) -> Result<(), ssl::Error> {
        Self::clear_error();
        Pin::new(&mut self.ssl).connect().await?;
        self.timing.established_ts = SystemTime::now();
        self.digest = Some(Arc::new(SslDigest::from_ssl(self.ssl())));
        Ok(())
    }


    /// Finish the TLS handshake from client as a server
    pub async fn accept(&mut self) -> Result<(), ssl::Error> {
        Self::clear_error();
        Pin::new(&mut self.ssl).accept().await?;
        self.timing.established_ts = SystemTime::now();
        self.digest = Some(Arc::new(SslDigest::from_ssl(self.ssl())));
        Ok(())
    }


    #[inline]
    fn clear_error() {
        let errs = tls::error::ErrorStack::get();
        if !errs.errors().is_empty() {
            warn!("Clearing dirty TLS error stack: {}", errs);
        }
    }
}

嘛，看到这里基本都看完了，很神奇吧）

Previous Post
Leaving Bytedance
Next Post
谈谈 tokio::select! 的公平性