Existence of Solutions to Fractional p-Laplacian Systems with Homogeneous Nonlinearities of Critical Sobolev Growth

In this paper, we investigate the existence of nontrivial solutions to the following fractional p-Laplacian system with homogeneous nonlinearities of critical Sobolev growth:\left\{\begin{aligned} \displaystyle{}(-\Delta_{p})^{s}u&\displaystyle=Q_{u}(u% ,v)+H_{u}(u,v)&&\displaystyle\phantom{}\text{in }\Omega,\\ \displaystyle(-\Delta_{p})^{s}v&\displaystyle=Q_{v}(u,v)+H_{v}(u,v)&&% \displaystyle\phantom{}\text{in }\Omega,\\ \displaystyle u=v&\displaystyle=0&&\displaystyle\phantom{}\text{in }\mathbb{R}% ^{N}\setminus\Omega,\\ \displaystyle u,v&\displaystyle\geq 0,\quad u,v\neq 0&&\displaystyle\phantom{}% \text{in }\Omega,\end{aligned}\right.where {(-\Delta_{p})^{s}} denotes the fractional p-Laplacian operator, {p>1}, {s\in(0,1)}, {ps<N}, {p_{s}^{*}=\frac{Np}{N-ps}} is the critical Sobolev exponent, Ω is a bounded domain in {\mathbb{R}^{N}} with Lipschitz boundary, and Q and H are homogeneous functions of degrees p and q with {p<q\leq p^{\ast}_{s}} and {Q_{u}} and {Q_{v}} are the partial derivatives with respect to u and v, respectively. To establish our existence result, we need to prove a concentration-compactness principle associated with the fractional p-Laplacian system for the fractional order Sobolev spaces in bounded domains which is significantly more difficult to prove than in the case of single fractional p-Laplacian equation and is of its independent interest (see Lemma 5.1). Our existence results can be regarded as an extension and improvement of those corresponding ones both for the nonlinear system of classical p-Laplacian operators (i.e., {s=1}) and for the single fractional p-Laplacian operator in the literature. Even a special case of our main results on systems of fractional Laplacian {(-\Delta)^{s}} (i.e., {p=2} and {0<s<1}) has not been studied in the literature before.