io_uring: fix race between poll completion and cancel_hash insertion

[ Upstream commit bd99c71bd14072ce2920f6d0c2fe43df072c653c ]

If poll arming and poll completion runs in parallel, there maybe races.
For instance, run io_poll_add in iowq and io_poll_task_func in original
context, then:

  iowq                                      original context
  io_poll_add
    vfs_poll
     (interruption happens
      tw queued to original
      context)                              io_poll_task_func
                                              generate cqe
                                              del from cancel_hash[]
    if !poll.done
      insert to cancel_hash[]

The entry left in cancel_hash[], similar case for fast poll.
Fix it by set poll.done = true when del from cancel_hash[].

Fixes: 5082620fb2ca ("io_uring: terminate multishot poll for CQ ring overflow")
Signed-off-by: Hao Xu <haoxu@linux.alibaba.com>
Link: https://lore.kernel.org/r/20210922101238.7177-2-haoxu@linux.alibaba.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>

diff --git a/fs/io_uring.c b/fs/io_uring.c
index 754d59f734d84893db12fed78cd8b533015592a0..27a1c813f1e11c51f8c4bcbd9c239ad8a91c6867 100644 (file)
--- a/fs/io_uring.c
+++ b/fs/io_uring.c
@@ -4968,10 +4968,8 @@ static bool io_poll_complete(struct io_kiocb *req, __poll_t mask)
        }
        if (req->poll.events & EPOLLONESHOT)
                flags = 0;
-       if (!io_cqring_fill_event(ctx, req->user_data, error, flags)) {
-               req->poll.done = true;
+       if (!io_cqring_fill_event(ctx, req->user_data, error, flags))
                flags = 0;
-       }
        if (flags & IORING_CQE_F_MORE)
                ctx->cq_extra++;
 
@@ -4993,6 +4991,7 @@ static void io_poll_task_func(struct io_kiocb *req)
                if (done) {
                        io_poll_remove_double(req);
                        hash_del(&req->hash_node);
+                       req->poll.done = true;
                } else {
                        req->result = 0;
                        add_wait_queue(req->poll.head, &req->poll.wait);
@@ -5126,6 +5125,7 @@ static void io_async_task_func(struct io_kiocb *req)
 
        hash_del(&req->hash_node);
        io_poll_remove_double(req);
+       apoll->poll.done = true;
        spin_unlock_irq(&ctx->completion_lock);
 
        if (!READ_ONCE(apoll->poll.canceled))